Serbia and Kosovo made little progress in lowering tensions and normalizing ties as leaders from both countries met for EU-backed talks in Brussels on Tuesday.
"Regrettably, the leaders were unable to reach any agreement today. And I am afraid that we will be facing, maybe, I hope not, a critical situation," Borrell told the media, urging more talks "soon."
Borrell had mediated the talks between Serbian President Aleksandar Vucic and Kosovo Prime Minister Albin Kurti.
The EU is trying to normalize relations between the countries, 15 years after Kosovo declared independence from Serbia. Violence has flared in Serb-populated northern Kosovo in recent months.
Kosovo, which today is almost exclusively inhabited by ethnic Albanians, seceded from Serbia in 1999 with NATO support and declared itself independent in 2008, although it remains unrecognized by Serbia.
The talks in Brussels were a follow-up to previous meetings about an 11-point, EU-backed peace plan. The most recent was in North Macedonia in March when the sides reached a tentative deal on implementation measures that was short on details and deadlines.
The planned agreement envisages that Belgrade will not recognize Kosovo under international law, but will take note of the statehood of its former province.
In particular, Serbia should recognize Kosovo's passports, license plates and customs documents, which it has not done to date.
Kosovo, in turn, will be bound to institutionally secure the rights of ethnic Serbs in the country. A sticking point has been Kosovo's failure to create of an Association of Serb Municipalities to protect the interests and rights of Serbs in Kosovo.
Borrell suggested on Tuesday night that if further negotiations fail, either country would not accept the EU. Serbia has officially been a candidate country since 2012, and Kosovo is considered a potential candidate country.