This question has popped up a few times during our reading of Exodus. Here is a good explanation by Peter Enns as found in the NIV Application Commentary.
"Without wading through the interpretive muck and mire, let me suggest that these verses presage the importance of circumcision in 12:43-49. We are able to make several fairly safe assumptions: (1) The object of God's wrath is Moses; no one really disputes this, and it makes the most sense in the context. (2) God is probably angry with Moses because his son (or perhaps both he and his son) is not circumcised. In any event, circumcision has something to do with it. (3) Zipporah's circumcision of her son appeases God's wrath and causes him to relent.
-----
What we see in 4:24-26 is the zeal with which the Lord guards this most important rite (circumcision). We must remember that circumcision as a sign of God's covenant was commanded of the patriarch Abraham (Gen. 17:1-27). Throughout the opening chapters of Exodus we have noted the importance God attaches to the patriarchs, particularly in connection with God's name (Ex 3:15-16; 4:5). It is in his connection to the patriarchs that God is to be known, and it is for the sake of the patriarchs that God will deliver Israel. Yet this connection to the patriarchs also imposes a covenant obligation on Moses and the Israelites. They have been delivered by the God of Abraham, Isaac, and Jacob. They are, therefore, to observe the command given to them: Circumcise your male children." ~ Peter Enns
