In late March 1938, the Japanese wished to solidify their hold on northern China, and to do this, they needed to take the city of Xuzhou and the area around it. Xuzhou lay between the Jiangsu province on the coast (which contained Shanghai and Nanjing), the Henan province to the west, where there were sizable Communist forces, and lay astride an important railway and canal system linking the north to the south. Leading into the populous city of Xuzhou was the smaller city of Tai'erzhuang. The city itself lay on the eastern side of the Grand Canal (the ancient and important waterway that transported many of the goods of eastern China), and it was also a key point along an important local railway. The control of Tai'erzhuang was important to both sides. For the Japanese, control of the area would allow them to move in many directions, including in the direction of the city of Wuhan, then a hub of Nationalist activity and troop concentrations.

There were a number of battles between 24th March - 7th April 1938. The Japanese army had a tendency to take matters into their own hands, and they often avoided getting permission from the Japanese government to continue the war. So, they marched into Jiangsu. The Chinese resistance was strong, and in April of 1938, the Chinese confronted the Japanese at Taierzhuang, which was located on the Grand Canal. Much to the surprise of the headstrong Japanese troops, they were beaten, making this battle the first major Chinese victory of the war.

By April 3rd, the Japanese troops in the city had been cornered, and the repeated attempts by Japanese units to break through and relieve the city had failed. By April 6th, 1938, what Japanese forces remained in the city were taken prisoner (nearly 800), and the Chinese got their hands on significant numbers of tanks and artillery, which would aid them in the time to come. The Battle of Tai'erzhuang, which officially ended on April 7th, was a shock to the Japanese, who had firmly believed they were invincible. In Japan itself, great effort was taken to keep the news of the defeat out of the press, and most Japanese never heard of it-even within parts of the military. For the Chinese, the victory at Tai'erzhuang was something they could rally around. It showed that their soldiers were capable of beating the Japanese and that their generals, at least some of them, were capable of leading men in battle.