Julio Ferracini • Product Design • deliverIt app • A/B Test
SEARCH AND PRESS ENTER

DeliverIt

  • Year 2017
  • Role UX Designer Consultant

Through the platform, there is no need for the end-user to use the app to fulfill the order. Through any channel (phone, website or app of the restaurant itself) the customer requests the order.

Therefore, the platform is geared towards winning partners interested in using its qualified delivery base, 24/7. Beta version.

Teste Goal:

Make the app more attractive and increase the base of registered restaurants by 50%.

About A/B test

The test consists of the release of different versions of the same interaction interface in order to evaluate the best performance between 2 options.

AB Test
AB Test

Criteria

In our case, two versions of the Legal Entity Registration screen will be released.

Version A – New version, clean and with highlighted call-to-action, without forms, the secondary colors of the brand predominate (green).

Version B – Original version, warm and relaxed, with a background alluding to the delivery system. The form on the first screen.

Hypothesis

Today the dropout rate on the PJ registration form is high compared to the current conversion: around 20% in terms of access.

1. The test recommendation is a sample of 280 users.

2. Statistical significance would be set at 95% in order to mitigate chance.

AB Test

Experimentation

ab-test

Key product performance indicators

Brand Consistency

How many times the application is shown in the search results on different platforms and social media.

Indicate improvements in keywords, in addition to redirecting engagement campaigns on social networks.

Measurement of user interest

A separate number of downloads, installations, and removal of the app for each version made available, a campaign carried out and platform.

In this way, measure the primary and secondary impact of the releases delivered.

App home page dropout rate

The number of users who do not start using the app because they have not passed the first screen.

Elaborate actions to mitigate the withdrawal of conquered users without the minimum deepening of the product proposal.

Navigation time and tasks

In particular, forms, seek improvement in the implementation of forms that encourage the enrichment of the base of companies available on the app.

A funnel to measure the average time spent in each section until the completion of the main activities.

Withdrawal rate

Check for withdrawals in key tasks of the app: PF registration and order completion (end-user).

Search and navigation

Metric to measure the efficiency of the app organization (Information Architecture), specifically on the Dashboard, as the time spent searching for the desired (and sometimes unknown) items decreases the conversion rate

Test overview

DeliverIt app / A/B test 1

Conclusions

A brief summary of the conclusions.

Determined result

We use the Google Optimizer tool to calculate the result after implementation. After the performance showed a big difference between the versions, we feel free to end the test after 185 sessions with a probability of 94% of the new version being the best option.

AB Test
AB Test

Decisions

In addition to changes to the Landing Page itself, it was defined that one of the main actions was to try to understand if the customer had real knowledge of the app’s purpose. After this test the recommendation to conduct qualitative research before launch for business refinement.

AB Test