Some confusions about Cosmology
The "Expanding" infinite Universe
We poor mortals think if something expands it has to be a finite object and expand into empty space. The Universe being everything does not have this option whether it is finite or infinite. "Expanding' to the Universe means if you chose any two points VERY (say, a billion or so light years) far apart. They will get further away from each other over time (and have been getting further away from each other from the Big Bang start). The Hubble constant - which is not really constant because of the interplay of gravity and dark energy - is the measure of the rate that distance increases.
Why do we suspect the Universe is infinite? Whether the universe is finite or infinite depends on the curvature of space that depends on the density of matter, dark matter and dark energy. Inflation implies a flat universe and thus it must be infinite. Direct measures of curvature are very close to flat.
How the Big Bang comes from nothing
You can choose your preference here.
1) We have some really good theories about what happens from the Planck time 10-43 seconds until now. That and all the evidence to support it give a good description of the Universe. So don't sweat the beginning. Since we know it looks like a singularity we have good cause to let it be what it is.
2) We can try to get some better physics for 0 to 10-43 seconds but we need to be okay with a start from 0. After all, that is a philosophical question.
3) The Universe we measure is part of a Multiverse that is infinite in time so the Big Bang was just a very spectacular transition. Multiverse theories then worry about how unique or special our universe is or what we can prove about the condition of the Multiverse before the Big Bang. There is some very fun physics in this field.
The Hubble expansion does not affect gravitationally bound systems. The current, as well as early Universe, expansion overcomes gravity. Systems such as a galaxy, a solar system, the Earth, have more gravity than the expansion can 'expand apart'. One can envision a universe coming into being where gravity was so large that it collapsed back to it singularity.
Some things that used to be speculation about cosmology we now have observational evidence for. Example: The large amount of H and He and the small amount of other elements implies the heavier elements were all formed in stars. We can get a good estimate of how much of the heavier ones are formed over the life of the Universe and it about matches what we observe. We can also find stars that are very old and they have almost nothing but H and He. We also look back into the furthest (and thus youngest) galaxies and we find only traces of the heavier elements. Armed with this evidence we can create an early universe scenario that matches a beginning with just H and He.