Heudorf Polenlager - my birth place in Germany; Polish parents

I was born in Germany in 1947 to Polish parents. My birth certificate has place of birth as Heudorf Polenlager. Through my research I have determined that Heudorf is Heudorf am Bussen in Bad Wurtemburg in Southern Germany. After the war the Castle in Heudorf (Schloss Heudorf) housed at one time 300 Poles. I am unable to find any further information on this. Heudorf is not even listed as a displaced persons camp in some of the dp camp lists I have found. I am looking for any record of my fathers side of the family (my grandfather, grandmother, aunts and uncles) who lived there as well as my my mother (she was the only one from her side of the family). I have old photographs of my mother and my aunt standing in front of the castle which match new photographs I obtained from the Heudorf am Bussen web site. We immigrated to the US in 1950 from Bremerhaven arriving at Ellis Island. I have found the ship passenger list with the names of my mother, father, brother and me. If anyone has any idea of where I can go to obtain more information please let me know. I have tried several ancestry web sites with no luck. My parents are both deceeased as well as my uncles and aunts and so I have no where to turn to get information. They did not want to talk much about their experience in Germany except for my mother who I know worked on a German farm after the Nazis took her from Poland when she was 14. My father was in the Polish army but I'm trying to figure out how most of his side of the family wound up in Heudorf after the war.

Have you had a look at the naturalization records of your parents yet? Your paternal grandfather should be listed there as well as his wife.

Regarding the labor camp, here's a German web site that might help you:
deutsche-digitale-bibliothek.de/item/XT4HNMDH73W53F4VLQZPAHKFNX6GC3LE

Could perhaps a local Hall of Records (Standesamt), disclosing your birth as well as the marriage of your parents, be of help?
Often in circumstances such as yours, it's usual for such records to go missing in all the hubbub and confusion, particularly with borders changing hands and so forth.