Pipeline dialog system was replaced by end-to-end methods, since neural networks methods become more and more popular.

DSTC (dialog system tracking challenge)

The Dialog State Tracking Challenge series-A Review introduced the first shared testbed and evaluation metrics

DSTC1 used a corpus of dialogs with various systems that participated in the Spoken Dialog Challenge (SDC) (Black et al., 2010), provided by the Dialog Research Center at Carnegie Mellon University. In the SDC, telephone calls from real passengers of the Port Authority of Allegheny County, which runs city buses in Pittsburgh, were forwarded to dialog systems built by different research groups. The goal was to provide bus riders with bus timetable information. For example, a caller might want to find out the time of the next bus leaving from Downtown to the airport. In this domain, the goal of the user typically remains fixed for the duration of the dialog.

DSTC2 aimed to extend the results of DSTC1 to another domain, as well as broaden the scope to include user goal changes. This challenge relied on a corpus of dialogs in the restaurant search domain between paid participants (through Amazon Mechanical Turk) and various systems developed at Cambridge University (Young et al., 2014). The goal of the user is to find specific information such as price range or phone number about a restaurant that fulfills a number of constraints such as cuisine or neighborhood.

DSTC3 expanded the domain of DSTC2 to include new slots which do not occur in the training data. This simulates the crucial problem of adapting a dialog system to a new domain for which little dialog data is available, while data for a similar but different domain might already exist. DSTC3 used all the data from DSTC2 as training set, as well as a new set of dialogs (also collected by Cambridge University researchers (Jurcˇ ́ıcˇek et al., 2011)) on a broader tourist information domain, covering bars and cafes in addition to restaurants.

[DSTC7][2]

Reference [1]: The Dialog State Tracking Challenge Series: A Review [2]: http://workshop.colips.org/dstc7/workshop.html