Challenge 2021

Sussex-Huawei Locomotion Challenge 2021

The Sussex-Huawei Locomotion Dataset [1-2] will be used in an activity recognition challenge with results to be presented at the HASCA Workshop at Ubicomp 2021.

This fourth edition of the challenge follows on our very successful 2018, 2019, and 2020 challenges, which saw the participation of more than 90 teams and 200 researchers [3-6].

The goal of this year edition is to recognize 8 modes of locomotion and transportation (activities) in a user-independent manner based on radio-data, including GPS reception, GPS location, Wifi reception and GSM cell tower scans. This is different from the previous three years that aimed at transportation mode recognition from the motion sensors.

This webpage will point to the data for the train and test dataset.

The participants will have to develop an algorithm pipeline that will process the sensor data, create models and output the recognized activities.

Prizes

  1. 800£
  2. 400£
  3. 200£

*Note: Prizes may increase subject to additional sponsors.

Deadlines

  • Registration via email: as soon as possible, but not later than 30.04.2021
  • Challenge duration: 13.04.2021 – 05.06.2021
  • Submission deadline:05.06.2021
  • HASCA-SHL paper submission: 15.06.2021
  • HASCA-SHL review notification: 15.07.2021
  • HASCA-SHL camera ready submission: 31.07.2021
  • HASCA workshop: 25.09.2021
  • Release of the ground-truth of the test data: 21.12.2023

Registration

Each team should send a registration email to shldataset.challenge@gmail.com as soon as possible but not later than 30.04.2021, stating the:

  • The name of the team
  • The names of the participants in the team
  • The organization/company (individuals are also encouraged)
  • The contact person with email address

HASCA Workshop

To be part of the final ranking, participants will be required to submit a detailed paper to the HASCA workshop. The paper should contain technical description of the processing pipeline, the algorithms and the results achieved during the development/training phase. The submissions must follow the HASCA format, but with a page limit between 3 and 6 pages.

Submission of predictions on the test dataset

The participants should submit a plain text prediction file (e.g. “teamName_predictions.txt) for the testing dataset, containing the time stamps and the predicted labels. Specifically, the submitted file should contain a matrix of size 671172 lines x 2 columns (the first column corresponds to the time stamps, and the second column corresponds to the prediction).

An example of submission is available here.

The participants’ predictions should be submitted online by sending an email to shldataset.challenge@gmail.com, in which there should be a link to the predictions file, using services such as Dropbox, Google Drive, etc. In case the participants cannot provide links using some file sharing service, they should contact the organizers via email shldataset.challenge@gmail.com, which will provide an alternate way to send the data.

To be part of the final ranking, participants will be required to publish a detailed paper in the proceedings of the HASCA workshop. The date for the paper submission is 15.06.2021. All the papers must be formatted using the ACM SIGCHI Master Article template with 2 columns. The template is available at TEMPLATES ISWC/UBICOMP2021. Submissions do not need to be anonymous.

Submission is electronic, using precision submission system. The submission site is open at https://new.precisionconference.com/submissions (select SIGCHI / UbiComp 2021 / UbiComp 2021 Workshop – HASCA-SHL and push Go button). See the image below.

A single submission is allowed per team. The same person cannot be in multiple teams, except if that person is a supervisor. The number of supervisors is limited to 3 per team.

Dataset format

The data is divided into three parts: train, validate and test. The data comprises of 59 days of training data, 6 days of validation data and 39 days of test data. The training data is collected by User1. The validation data is a mixture of User2 and User3. The testing data is a mixture of User2 and User3.

Note: All the sensors are asynchronously sampled. The sampling rate is roughly 1 Hz, but is time-varying for each sensor. Note that, depending on the condition of the satellite and cell, it may happen that one sensor receive no signal at a certain interval and thus no data recorded. The time data at the first column of each sensor contains the epoch time in ms.

The training data is collected by User 1 with a phone at the Hips location in 59 days. The format of the data is shown in the table below.

Filename Size Format
Column Content
Location.txt 911109 x n_var 1 Epoch time [ms]
2 Ignore
3 Ignore
4 Accuracy of this location [m]
5 Latitude [degrees]
6 Longitude [degrees]
7 Altitude [m]
GPS.txt 1322749 x n_var 1 Epoch time [ms]
2 Ignore
3 Ignore
4+

Variable number of entries for GPS data. If no satellite is visible the 4th column is 0. Otherwise, for each satellite visible 4 columns are added to the data file and an additional last column indicates the number of satellites. Each of the 4 columns contain in order: ID, SNR, Azimuth [degrees], Elevation [degrees]

For example:

1489485950011 161777247369 10889909374 0
indicates no satellite visible.

1489485951014 162780045286 10889909374 7 12.0 56.0 32.0 1
indicates one satellite visible; satellite 7 with SNR=12, Azimuth=56 and elevation=32.

1489485962025 173791715076 10889909374 7 15.0 56.0 32.0 30 12.0 82.0 70.0 2
indicates two satellite visible; satellite 7 and 30.
WiFi.txt 1459351 x n_var 1 Epoch time [ms]
2 Ignore
3 Ignore
4+

Variable number of Wifi data. For each visible Wifi 5 semicolon delimited fields are included, in order: BSSID, SSID, RSSI, Frequency [MHz], Capabilities.
Cells.txt 1324881 x n_var 1 Epoch time [ms]
2 Ignore
3 Ignore
4 Number of entries
5+

Variable number of fields depending on entries. The first field identifies the type of cell: LTE, WCDMA or GSM. The number of fields depends on the cell type and is as follows:

LTE; isRegistered; ci; MCC; MNC; PCI; TAC; asuLevel; dBm; level.

isRegistered status of the connection of the phone to this cell, 1=connected, 0=not connected
ci 28-bit Cell Identity
MCC 3-digit Mobile Country Code
MNC 2 or 3-digit Mobile Network Code
PCI Physical Cell Id
TAC 16-bit Tracking Area Code
asuLevel signal level based on on 3GPP RSRP
dBm signal strength in dBm
level signal level as int between 0 and 4.

GSM; isRegistered; cid; lac; MCC; MNC; asuLevel; dBm; level;

isRegistered status of the connection of the phone to this cell, 1=connected, 0=not connected
cid 16-bit GSM Cell Identity described in TS 27.007
lac 160bit Location Area Code
MCC 3-digit Mobile Country Code
MNC 2 or 3-digit Mobile Network Code
asuLevel signal level based on on 3GPP RSRP
dBm signal strength in dBm
level signal level as int between 0 and 4.

WCDMA; isRegistered; cid; lac; MCC; MNC; PSC; asuLevel; dBm; level

isRegistered status of the connection of the phone to this cell, 1=connected, 0=not connected
cid 16-bit GSM Cell Identity described in TS 27.007
lac 160bit Location Area Code
MCC 3-digit Mobile Country Code
MNC 2 or 3-digit Mobile Network Code
PSC 9-bit UMTS Primary Scrambling Code described in TS 25.331
asuLevel signal level based on on 3GPP RSRP
dBm signal strength in dBm
level signal level as int between 0 and 4.
Label.txt 980527 x 2 1 Epoch time [ms]
2 Label: Still=1, Walking=2, Run=3, Bike=4, Car=5, Bus=6, Train=7, Subway=8

The validation data is collected by User2 and User3 with a phone at the Hips location in 4 days. The format of the data is shown in the table below.

Filename Size Format
Location.txt 101524 x n_var The same as the training data.
GPS.txt 157348 x n_var The same as the training data.
WiFi.txt 158681 x n_var The same as the training data.
Cells.txt 126333 x n_var The same as the training data.
Label.txt 143958 x 2 The same as the training data.

The testing data is collected by User2 and User3 with a phone at the Hips location in 39 days. The format of the data is shown in the table below.

Filename Size Format
Location.txt 562565 x n_var The same as the training data.
GPS.txt 781773 x n_var The same as the training data.
WiFi.txt 771855 x n_var The same as the training data.
Cells.txt 561369 x n_var The same as the training data.
Label_idx.txt 671172 x 1 The timestamps for which predict the transportation mode.

Downloads

Data

Submission example

This is an example of how the submission classification result should look like: challenge-2021-example_submission.

Ground truth of the test data (released on 21/12/2023)

Download the test labels of the test set.

Rules

Some of the main rules are listed below. The detailed rules are contained in the following document.

  • Eligibility
    • You do not work in or collaborate with the SHL project (http://www.shl-dataset.org/);
    • If you submit an entry, but are not qualified to enter the contest, this entry is voluntary. The organizers reserve the right to evaluate it for scientific purposes. If you are not qualified to submit a contest entry and still choose to submit one, under no circumstances will such entries qualify for sponsored prizes.
  • Entry
    • Registration (see above): as soon as possible but not later than 15.04.2021.
    • Challenge: Participants will submit prediction results on test data.
    • Workshop paper: To be part of the final ranking, participants will be required to publish a detailed paper in the proceedings of the HASCA workshop (http://hasca2021.hasc.jp/); The dates will be set during the competition.
    • Submission: The participants’ predictions should be submitted online by sending an email to shldataset.challenge@gmail.com, in which there should be a link to the predictions file, using services such as Dropbox, Google Drive, etc. In case the participants cannot provide link using some file sharing service, they should contact the organizers via email shldataset.challenge@gmail.com, which will provide an alternate way to send the data.
    • A single submission is allowed per team. The same person cannot be in multiple teams, except if that person is a supervisor. The number of supervisors is limited to 3 per team.

Q&A

Contact

All inquiries should be directed to: shldataset.challenge@gmail.com

Organizers

  • Prof. Daniel Roggen, University of Sussex (UK)
  • Prof. Hristijan Gjoreski, University of Sussex (UK) & Ss. Cyril and Methodius University (MK)
  • Dr. Lin Wang, Queen Mary University of London (UK)
  • Mathias Ciliberto, University of Sussex (UK)
  • Dr. Kazuya Murao, Ritsumeikan University (JP)
  • Dr. Tsuyoshi Okita, Kyushu Institute of Technology (JP)
  • Dr. Paula Lago, Universidad Nacional Abierta y a Distancia (CO)

References

[1] H. Gjoreski, M. Ciliberto, L. Wang, F.J.O. Morales, S. Mekki, S. Valentin, and D. Roggen, “The University of Sussex-Huawei locomotion and transportation dataset for multimodal analytics with mobile devices,” IEEE Access 6 (2018): 42592-42604. [DATASET INTRODUCTION]

[2] L. Wang, H. Gjoreski, M. Ciliberto, S. Mekki, S. Valentin, and D. Roggen, “Enabling reproducible research in sensor-based transportation mode recognition with the Sussex-Huawei dataset,” IEEE Access 7 (2019): 10870-10891. [GPS BASELINE + DATASET ANALYSIS ]

[3] L. Wang, H. Gjoreski, M. Ciliberto, S. Mekki, S. Valentin, and D. Roggen, “Benchmarking the SHL recognition challenge with classical and deep-learning pipelines,” in Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pp. 1626-1635, 2018. [BASELINE FOR MOTION SENSORS]

[4] L. Wang, H. Gjoreski, K. Murao, T.  Okita, and D.l Roggen, “Summary of the Sussex-Huawei locomotion-transportation recognition challenge,” in Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pp. 1521-1530, 2018. [SHL 2018 SUMMARY]

[5] L. Wang, H. Gjoreski, M. Ciliberto, P. Lago, K. Murao, T.  Okita, and D. Roggen, “Summary of the Sussex-Huawei locomotion-transportation recognition challenge 2019,” in Proceedings of the 2019 ACM International Joint Conference and 2019 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pp. 849-856, 2019. [SHL 2019 SUMMARY]

[6] L. Wang, H. Gjoreski, M. Ciliberto, P. Lago, K. Murao, T.  Okita, and D. Roggen, “Summary of the Sussex-Huawei locomotion-transportation recognition challenge 2020,” in Proceedings of the 2020 ACM International Joint Conference and 2020 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pp. 849-856, 2020. [SHL 2020 SUMMARY]

 

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

Competition Results

Summary paper:

Locomotion and transportation Mode Recognition from GPS and radio signals: Summary of SHL Challenge 2021 
Lin Wang, Mathias Ciliberto, Hristijan Gjoreski, Paula Lago, Kazuya Murao, Tsuyoshi Okita, Daniel Roggen.
[PDF]

 

Winning teams: 

  • DD (75.40%)
  • Guerrilla (74.26%)
  • We_can_fly (63.28%)