In addition to dragons, another sort of being that shows up in just about every known myth cycle is the giant. In Norse mythology, the frost giants are the enemies of the gods. The Greeks had the earth and sky give birth to hundred-armed giants and cyclopes before producing the Titans, who were generally considered to be pretty big themselves. The Hindus had the Daityas, who were (you guessed it) enemies of the Deva. Paul Bunyan is a giant of varying size (sometimes big enough to carve the Grand Canyon single-handedly, but other times small enough to interact with normal humans) from American tall tales. Fairy tales have a lot of giants, typically quite ravenous and dumb, and pretty much inevitably killed off by plucky peasant boys named Jack. The Irish Finn MacCool (more accurately referred to as Fionn mac Cumhaill, but the anglicized version has to be the ultimate eighties name) was a heroic giant. The folklore of the British Isles also made the enigmatic Biblical figures Gog and Magog into giants, sometimes portrayed as evil figures slain by early settlers of the Isles, and other times as the protectors of the City of London. And speaking of the Bible, it has the Nephilim (giant offspring of humans and angels) in Genesis, and King Og of Bashan in Numbers. Later on, the giant Philistine warrior Goliath shows up just in time to be killed by David. As popular as this tale is, though, 2 Samuel has it that Goliath was killed not by the legendary king, harpist, psalmist, and wife-snatcher; but by the more obscure Elhanan. Sounds like David might well have stolen credit for heroic deeds as well as Hittites' wives.
Goliath, at a height of six cubits and a span, was unlikely to have been the height of Og, who was said to have slept in a thirteen-foot-long bed (although it's possible Og just needed a lot of room to toss and turn, or to fit in multiple concubines). The even earlier Nephilim are described in the Book of Enoch (which, admittedly, was written after the stories of Goliath, but might possibly have been based on earlier legends) as either 30 or three hundred cubits tall. Just like human lifespans became more believable over the course of the Bible, so did the size of giants. And really, it's pretty common in mythology to place the heyday of the huge giants in the distant past, with modern giants being either much smaller or living in isolated regions.
So where did the concept of giants arise? Some of the stories might have been based on enormous bones, like how the cyclopes are said to have been inspired by elephant skulls, with the space for the trunk being thought to hold a giant eye instead. And some theories had it that the more impressive of the ancient cities HAD to have been built by giants. In general, though, I think the idea of people who were really big is a pretty basic exercise in imagination. We now know that the giants of really enormous proportions wouldn't have been able to walk upright, but I try to give a bit of leeway when tales of giants are involved.