ipynb • Lines: 1571{
"cells": [
{
"cell_type": "markdown",
"id": "751a404c",
"metadata": {},
"source": [
"# AB Testing\n",
"\n",
"Welcome! In this assignment you will be presented with two cases that require an AB test to choose an action to improve an existing product. You will perform AB test for a continuous and a proportion metric. For this you will define functions that estimate the relevant information out of the samples, compute the relevant statistic given each case and make a decision on whether to (or not) reject the null hypothesis.\n",
"\n",
"Let's get started!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "a5759717",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"import math\n",
"import numpy as np\n",
"import pandas as pd\n",
"import scipy.stats as stats\n",
"from dataclasses import dataclass\n",
"import utils"
]
},
{
"cell_type": "markdown",
"id": "ab8f62a7",
"metadata": {},
"source": [
"# Section 1: Continuous Metric - Average Session Duration\n",
"\n",
"Suppose you have a website that provides machine learning content in a blog-like format. Recently you saw an article claiming that similar websites could improve their engagement by simply using a specific color palette for the background. Since this change seems pretty easy to implement you decide to run an AB test to see if this change does in fact drive your users to stay more time in your website.\n",
"\n",
"The metric you decide to evaluate is the `average session duration`, which measures how much time on average your users are spending on your website. This metric currently has a value of 30.87 minutes.\n",
"\n",
"Without further considerations you decide to run the test for 20 days by randomly splitting your users into two segments:\n",
"- `control`: These users will keep seeing your original website.\n",
"\n",
"\n",
"- `variation`: These users will see your website with the new background colors.\n",
"\n",
"Run the next cell to load the data from the test:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9ebfb03e",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>user_id</th>\n",
" <th>user_type</th>\n",
" <th>session_duration</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>FDWYR0BDXL</td>\n",
" <td>variation</td>\n",
" <td>15.528769</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>ZQ9CJ46TZI</td>\n",
" <td>variation</td>\n",
" <td>32.287590</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>3HF7VF53CE</td>\n",
" <td>variation</td>\n",
" <td>43.718217</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>9ZEWSA8TZE</td>\n",
" <td>variation</td>\n",
" <td>49.519702</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>YXG0A4QWAE</td>\n",
" <td>control</td>\n",
" <td>61.709028</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5</th>\n",
" <td>UP81SFIYAR</td>\n",
" <td>variation</td>\n",
" <td>71.779283</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6</th>\n",
" <td>8HQYE1Z7ZJ</td>\n",
" <td>variation</td>\n",
" <td>23.291835</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>70XQO3TSB3</td>\n",
" <td>control</td>\n",
" <td>25.219461</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>F2WGCJ5NW6</td>\n",
" <td>control</td>\n",
" <td>26.240482</td>\n",
" </tr>\n",
" <tr>\n",
" <th>9</th>\n",
" <td>J00IM2JWLD</td>\n",
" <td>variation</td>\n",
" <td>20.780244</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" user_id user_type session_duration\n",
"0 FDWYR0BDXL variation 15.528769\n",
"1 ZQ9CJ46TZI variation 32.287590\n",
"2 3HF7VF53CE variation 43.718217\n",
"3 9ZEWSA8TZE variation 49.519702\n",
"4 YXG0A4QWAE control 61.709028\n",
"5 UP81SFIYAR variation 71.779283\n",
"6 8HQYE1Z7ZJ variation 23.291835\n",
"7 70XQO3TSB3 control 25.219461\n",
"8 F2WGCJ5NW6 control 26.240482\n",
"9 J00IM2JWLD variation 20.780244"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Load the data from the test\n",
"data = utils.run_ab_test_background_color(n_days=20)\n",
"\n",
"# Print the first 10 rows\n",
"data.head(10)"
]
},
{
"cell_type": "markdown",
"id": "2d257a0f",
"metadata": {},
"source": [
"The data shows for every user the average session duration and the version of the website they interacted with. To separate both segments for easier computations you can slice the dataframe by running the following cell:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "9552f1a8",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2069 users saw the original website with an average duration of 32.92 minutes\n",
"\n",
"2117 users saw the new website with an average duration of 33.83 minutes\n"
]
}
],
"source": [
"# Separate the data from the two groups (sd stands for session duration)\n",
"control_sd_data = data[data[\"user_type\"]==\"control\"][\"session_duration\"]\n",
"variation_sd_data = data[data[\"user_type\"]==\"variation\"][\"session_duration\"]\n",
"\n",
"print(f\"{len(control_sd_data)} users saw the original website with an average duration of {control_sd_data.mean():.2f} minutes\\n\")\n",
"print(f\"{len(variation_sd_data)} users saw the new website with an average duration of {variation_sd_data.mean():.2f} minutes\")"
]
},
{
"cell_type": "markdown",
"id": "0946faf2",
"metadata": {},
"source": [
"Notice that the split is not perfectly balanced. This is common in AB testing as there is randomness associated with the way the users are assigned to each group. \n",
"\n",
"At first glance it looks like the change to the background did in fact drive users to stay longer on your website. However you know better than driving conclusions at face value out of this data so you decide to perform a hypothesis test to know if there is a significant difference between the **means** of these two segments. You can do this by computing the t-statistic and using the null hypothesis that there is **not** a statistically significant difference between the means of the two samples:\n",
"\n",
"$$t = \\frac{(\\bar{x}_{1} - \\bar{x}_{2}) - (\\mu_1 - \\mu_2)}{\\sqrt{\\frac{s_{1}^2}{n_1} + \\frac{s_{2}^2}{n_2}}}$$\n",
"\n",
" \n",
"\n",
"Notice that by computing the metric at a user level you ensure that the independence criteria is met since each user is independent of one another. Also, although the data is not strictly normal you have a large enough sample size to justify the use of the t-test.\n",
"\n",
"Before finding the value of the statistic you will need to compute all the necessary metrics for every group. For this you decide to use a dataclass that holds this information:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "20e6f0fe",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"@dataclass\n",
"class estimation_metrics_cont:\n",
" n: int\n",
" xbar: float\n",
" s: float\n",
" \n",
" def __repr__(self):\n",
" return f\"sample_params(n={self.n}, xbar={self.xbar:.3f}, s={self.s:.3f})\""
]
},
{
"cell_type": "markdown",
"id": "55cd73dd",
"metadata": {},
"source": [
"This class will hold the information for $n$, $\\bar{x}$ and $s$. \n",
"\n",
"## Exercise 1: compute_continuous_metrics\n",
"\n",
"Now that you have a container for all these metrics, it is your job to code a function that given some data will compute them for that particular set of data. To do this complete the `compute_continuous_metrics` below.\n",
"\n",
"Hints:\n",
"- [np.mean](https://numpy.org/doc/stable/reference/generated/numpy.mean.html), [np.std](https://numpy.org/doc/stable/reference/generated/numpy.std.html) and [len](https://docs.python.org/3/library/functions.html#len) functions are useful.\n",
"- When computing the sample standard deviation be sure to use the $s$ estimation: $s=\\frac{1}{N-1}\\sum_{i=1}^n(x_i - \\overline x)^2$. To accomplish this you can set the parameter `ddof=1` within the [np.std](https://numpy.org/doc/stable/reference/generated/numpy.std.html) function. This ensures that the denominator of the expression for the standard deviation is `N-1` rather than `N`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "243ee36c",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def compute_continuous_metrics(data):\n",
" \"\"\"Computes the relevant metrics out of a sample for continuous data.\n",
"\n",
" Args:\n",
" data (pandas.core.series.Series): The sample data. In this case the average session duration for each user.\n",
"\n",
" Returns:\n",
" estimation_metrics_cont: The metrics saved in a dataclass instance.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" metrics = estimation_metrics_cont( \n",
" n=len(data),\n",
" xbar=np.mean(data),\n",
" s=np.std(data,ddof=1)\n",
" ) \n",
" ### END CODE HERE ###\n",
" \n",
" return metrics"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "49345865",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"n=5, xbar=3.00 and s=1.58 for example array\n",
"\n",
"n=2069, xbar=32.92 and s=17.54 for control data\n",
"\n",
"n=2117, xbar=33.83 and s=18.24 for variation data\n"
]
}
],
"source": [
"# Test your code\n",
"\n",
"cm = compute_continuous_metrics(np.array([1,2,3,4,5]))\n",
"print(f\"n={cm.n}, xbar={cm.xbar:.2f} and s={cm.s:.2f} for example array\\n\")\n",
"\n",
"control_metrics = compute_continuous_metrics(control_sd_data)\n",
"print(f\"n={control_metrics.n}, xbar={control_metrics.xbar:.2f} and s={control_metrics.s:.2f} for control data\\n\")\n",
"\n",
"variation_metrics = compute_continuous_metrics(variation_sd_data)\n",
"print(f\"n={variation_metrics.n}, xbar={variation_metrics.xbar:.2f} and s={variation_metrics.s:.2f} for variation data\")"
]
},
{
"cell_type": "markdown",
"id": "06048083",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"n=5, xbar=3.00 and s=1.58 for example array\n",
"\n",
"n=2069, xbar=32.92 and s=17.54 for control data\n",
"\n",
"n=2117, xbar=33.83 and s=18.24 for variation data\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "97312678",
"metadata": {},
"source": [
"## Exercise 2: degrees_of_freedom\n",
"\n",
"Another important piece of information when performing a t-test is the degrees of freedom, which can be computed as follows:\n",
"\n",
"$$\\text{Degrees of freedom } = \\frac{\\left[\\frac{s_{1}^2}{n_1} + \\frac{s_{2}^2}{n_2} \\right]^2}{\\frac{(s_{1}^2/n_1)^2}{n_1-1} + \\frac{(s_{2}^2/n_2)^2}{n_2-1}}$$\n",
"\n",
"Complete the `degrees_of_freedom` function below so that given two samples it will return the degrees of freedom of the underlying t-distribution of the test statistic. Notice that this value does not necessarily need to be an integer and can be a float. \n",
"\n",
"Hints:\n",
"- Use the `compute_continuous_metrics` function you previously coded to compute the metrics for each sample. \n",
"\n",
"<!-- --> \n",
"\n",
"- You can use [np.square](https://numpy.org/doc/stable/reference/generated/numpy.square.html) to get the square of a value.\n",
"\n",
"<!-- --> \n",
"\n",
"\n",
"- In this context the suffix 1 denotes the control data, while 2 denotes the variation data (this applies to all functions in this assignment).\n",
"\n",
"<!-- --> \n",
"\n",
"\n",
"- To retrieve information from the metrics dataclass you can use the dot (.) notation. For example if you have defined the following variable `control_metrics = compute_continuous_metrics(control_sample)`, you can get $s$ by using the expression `control_metrics.s`\n",
"\n",
"<!-- --> \n",
"\n",
"\n",
"- You can assign multiple values in Python in the same line. For example if you have a class with attributes $a$, $b$ and $c$ you can do something like `a1, b1, c1 = class.a, class.b, class.c`. This sometimes can make code easier to read."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a8d352bd",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def degrees_of_freedom(control_metrics, variation_metrics):\n",
" \"\"\"Computes the degrees of freedom for two samples.\n",
"\n",
" Args:\n",
" control_metrics (estimation_metrics_cont): The metrics for the control sample.\n",
" variation_metrics (estimation_metrics_cont): The metrics for the variation sample.\n",
"\n",
" Returns:\n",
" numpy.float: The degrees of freedom.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" \n",
" n1, s1 = control_metrics.n,control_metrics.s\n",
" n2, s2 = variation_metrics.n,variation_metrics.s\n",
" \n",
" \n",
" dof = np.square((s1**2 / n1 + s2**2 / n2)) / ((s1**2 / n1)**2 / (n1 - 1) + (s2**2 / n2)**2 / (n2 - 1))\n",
" \n",
" ### END CODE HERE ###\n",
" \n",
" \n",
" return dof"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "b29f42d8",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"DoF for example arrays: 2.88\n",
"\n",
"DoF for AB test samples: 4182.97\n"
]
}
],
"source": [
"# Test your code\n",
"test_m1, test_m2 = compute_continuous_metrics(np.array([1,2,3])), compute_continuous_metrics(np.array([4,5]))\n",
"dof = degrees_of_freedom(test_m1, test_m2)\n",
"print(f\"DoF for example arrays: {dof:.2f}\\n\")\n",
"\n",
"dof = degrees_of_freedom(control_metrics, variation_metrics)\n",
"print(f\"DoF for AB test samples: {dof:.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "3a8a6ac0",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"DoF for example arrays: 2.88\n",
"\n",
"DoF for AB test samples: 4182.97\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "aedfcc03",
"metadata": {},
"source": [
"## Exercise 3: t_statistic_diff_means\n",
"\n",
"Now you have everything you need to perform the hypothesis testing. Complete the `t_statistic_diff_means` which given two samples should return the t-statistic, which should be computed like this:\n",
"\n",
"$$t = \\frac{(\\bar{x}_{1} - \\bar{x}_{2}) - (\\mu_1 - \\mu_2)}{\\sqrt{\\frac{s_{1}^2}{n_1} + \\frac{s_{2}^2}{n_2}}}$$\n",
"\n",
"Hints:\n",
"- You can use [np.sqrt](https://numpy.org/doc/stable/reference/generated/numpy.sqrt.html) to compute the squared root of a value.\n",
"\n",
"<!-- --> \n",
"\n",
"- The value for the difference of $(\\mu_1 - \\mu_2)$ should be replaced by the value of this difference under the null hypothesis.\n",
"\n",
" \n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "d2ff3a70",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def t_statistic_diff_means(control_metrics, variation_metrics):\n",
" \"\"\"Compute the t-statistic for the difference of two means.\n",
"\n",
" Args:\n",
" control_metrics (estimation_metrics_cont): The metrics for the control sample.\n",
" variation_metrics (estimation_metrics_cont): The metrics for the variation sample.\n",
"\n",
" Returns:\n",
" numpy.float: The value of the t-statistic.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" \n",
" n1, xbar1, s1 = control_metrics.n, control_metrics.xbar, control_metrics.s\n",
" n2, xbar2, s2 = variation_metrics.n, variation_metrics.xbar, variation_metrics.s\n",
" \n",
" t = (xbar1 - xbar2) / np.sqrt((s1 ** 2) / n1 + (s2 ** 2) / n2)\n",
" ### END CODE HERE ###\n",
" \n",
" return t"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "6f455b84",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"t statistic for example arrays: -3.27\n",
"\n",
"t statistic for AB test: -1.64\n"
]
}
],
"source": [
"# Test your code\n",
"\n",
"t = t_statistic_diff_means(test_m1, test_m2)\n",
"print(f\"t statistic for example arrays: {t:.2f}\\n\")\n",
"\n",
"t = t_statistic_diff_means(control_metrics, variation_metrics)\n",
"print(f\"t statistic for AB test: {t:.2f}\")"
]
},
{
"cell_type": "markdown",
"id": "e978292c",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"t statistic for example arrays: -3.27\n",
"\n",
"t statistic for AB test: -1.64\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "9581ac98",
"metadata": {},
"source": [
"## Exercise 4: reject_nh_t_statistic\n",
"\n",
"With the ability to calculate the t-statistic now you need a way to determine if you should reject (or not) the null hypothesis. Complete the `reject_nh_t_statistic` function below. This function should return whether to reject (or not) the null hypothesis by using the *p*-value method given the value of the observed statistic, the degrees of freedom and a significance level. The way you have formulated the hypotheses, **this should be a two-sided test.**\n",
"\n",
"In this case the *p*-value represents the probability of obtaining a value of the t-statistic as extreme as or more extreme than the observed value under the null hypothesis that both means are the same. You can use the fact that the `CDF` of a distribution provides the probability of getting a value less than or equal to the one provided, so the probability that the statistic is greater than the observed value can be computed as `1 - CDF(observed_value)`. If you are conducting a two-sided test, then \"more extreme\" means that the absolute value of the statistic is greater than the absolute value of the observed statistic. In other words, \"more extreme\" happens when the statistic being too big or too small. Since the distribution under $H_0$ is symmetric around 0, the probabilities at each tail of the distribution are the same, and you can get the *p*-value by multiplying said `1-CDF(|observed_value|)` by 2, because the probability at both tails is the same.\n",
"\n",
"Hints:\n",
"- You can use the `cdf` method from the [stats.t](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.t.html) class to compute the p-value given the degrees of freedom. Don't forget to multiply your result by 2 since this is a two-sided test.\n",
"\n",
"<!-- --> \n",
"\n",
"\n",
"- When passing the value of the t-statistic to the `cdf` function, you should provide the absolute value. You can achieve this by using Python's built-in `abs` function.\n",
"\n",
"<!-- --> \n",
"\n",
"\n",
"- If the p-value is lower to `alpha` then you should reject the null hypothesis."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "187c065f",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def reject_nh_t_statistic(t_statistic, dof, alpha=0.05):\n",
" \"\"\"Decide whether to reject (or not) the null hypothesis of the t-test.\n",
"\n",
" Args:\n",
" t_statistic (numpy.float): The computed value of the t-statistic for the two samples.\n",
" dof (numpy.float): The computed degrees of freedom for the two samples.\n",
" alpha (float, optional): The desired significance level. Defaults to 0.05.\n",
"\n",
" Returns:\n",
" bool: True if the null hypothesis should be rejected. False otherwise.\n",
" \"\"\"\n",
" \n",
" reject = False\n",
" ### START CODE HERE ###\n",
" p_value = 2 * (1 - stats.t.cdf(abs(t_statistic), df=dof))\n",
" \n",
" if p_value < alpha:\n",
" reject = True\n",
" ### END CODE HERE ###\n",
" \n",
" return reject"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "b8ca86d8",
"metadata": {
"scrolled": true,
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The null hypothesis can be rejected at the 0.05 significance level: False\n",
"\n",
"There is not enough statistical evidence against H0.\n",
"It can be concluded that there is not a statistically significant difference between the means of the two samples.\n"
]
}
],
"source": [
"# Test your code\n",
"\n",
"alpha = 0.05\n",
"reject_nh = reject_nh_t_statistic(t, dof, alpha)\n",
"\n",
"print(f\"The null hypothesis can be rejected at the {alpha} significance level: {reject_nh}\\n\")\n",
"\n",
"msg = \"\" if reject_nh else \" not\"\n",
"print(f\"There is{msg} enough statistical evidence against H0.\\nIt can be concluded that there is{msg} a statistically significant difference between the means of the two samples.\")"
]
},
{
"cell_type": "markdown",
"id": "137c7516",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"The null hypothesis can be rejected at the 0.05 significance level: False\n",
"\n",
"There is not enough statistical evidence against H0.\n",
"It can be concluded that there is not a statistically significant difference between the means of the two samples.\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "6b543691",
"metadata": {},
"source": [
"Given the initial values for each group it looked like the change in the background could be having the positive impact it was initially thought. However after performing the hypothesis testing you can conclude that there is not enough statistical evidence to reject the null hypothesis at a significance level of 0.05, so you can't confirm that the average session duration was affected by the change and the slight increase you saw at first may be due to randomness."
]
},
{
"cell_type": "markdown",
"id": "b4d8aa51",
"metadata": {},
"source": [
"# Section 2: Proportions - Conversion Rate (CVR)"
]
},
{
"cell_type": "markdown",
"id": "7f34907d",
"metadata": {},
"source": [
"After the experience with your own website you decided to work as a full time Data Analyst helping other companies run their AB tests. Currently you are working for a food delivery app to determine if a new feature (which provides custom suggestions to each user based on their preferences) will increase the `conversion rate` of the app. This rate measures the rate of users who \"converted\" or placed an order using the app. By now you know that most companies use proportion-based metrics to measure their AB tests since these are typically well understood by stakeholders and an economic value is usually predefined for them.\n",
"\n",
"\n",
"One thing that is commonly done when performing AB testing, is to take into account the sample size required to get a desired performance. As you already saw, independently of the number of samples available you will always achieve your significance level goal, since you design the test for that. However, you might also want to set some goals for your type II error. Recall that for given a significance level and number of samples, type II error is fixed, but you could always improve it by geting more samples. The question is, how many samples do you need?. You will learn how to calculate that on this section of the assignent. \n",
"\n",
"\n",
"#### Sample size needed to compare two binomial proportions using a two-sided test with significance level $\\alpha$ and power $1 - \\beta$ where one sample $(n_2)$ is $k$ times as large as the other sample $(n_1)$ (independent-sample case)\n",
"\n",
"To test the hypothesis $H_0:p_1 = p_2$ vs. $H_1: p_1 \\neq p_2$ with significance level $\\alpha$, and a power $1 - \\beta$ at the specific alternative $\\mid p_1 - p_2 \\mid = \\Delta$, the following sample size is required\n",
"\n",
"$$n_1 = \\frac{\\left[\\sqrt{\\bar{p}\\bar{q}\\left(1 + \\frac{1}{k} \\right)}z_{1- \\alpha/2} + \\sqrt{p_1 q_1 + \\frac{p_2q_2}{k}}z_{1-\\beta}\\right]^2}{\\Delta^2}$$\n",
"$$n_2 = k n_1$$\n",
"where $p_1,p_2 = $ projected true probabilities of success in the two groups and\n",
"$$q_1,q_2 = 1 - p_1, 1 - p_2$$\n",
"$$\\Delta = \\mid p_2 - p_1 \\mid$$\n",
"$$\\overline{p} = \\frac{p_1 + kp_2}{1+ k}$$\n",
"\n",
"\n",
"Going back to your sales problem, **the current CVR of the app is 12% and the stakeholders would like the new feature to increase it up to a 14%**. Given this expectation you can compute the required sample size using the above formula. In this case $p_1$ would be 0.12, and $p_2$ 0.14. $\\beta$, which is the type II error at $p_1-p_2 = 0.02$, should be some small value, lets say for example 0.2.\n",
"\n",
"The function to perform this calculation is already provided for you and can be determined by running the following cell:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "93894ca8",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"data": {
"text/plain": [
"4438"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Compute the sample size required to compare the actual vs desired CVR\n",
"required_sample_size = utils.sample_size_diff_proportions(0.12, 0.14)\n",
"required_sample_size"
]
},
{
"cell_type": "markdown",
"id": "88adc638",
"metadata": {},
"source": [
"You would need around 4400 users per group to be able to detect a difference between the current CVR and the expected one with a significance level of 0.05 and a power of 0.8.\n"
]
},
{
"cell_type": "markdown",
"id": "e4df2211",
"metadata": {},
"source": [
"In case you are wondering about this computation but for the continuos metrics case (section 1). Click the robot to see the formula:\n",
"\n",
"<details>\n",
" <summary>➤ 🤖</summary>\n",
" \n",
"#### Sample size needed for comparing the means of two normally distributed samples of equal size using a two-sided test with significance level $\\alpha$ and power $1 - \\beta$\n",
"\n",
"$$ n = \\frac{\\left(\\sigma_{1}^{2} + \\sigma_{2}^2 \\right) \\left(z_{1-\\alpha/2} + z_{1-\\beta} \\right)^2}{\\Delta^2} = \\text{sample size for each group}$$\n",
"\n",
"where $\\Delta = \\mid \\mu_{2} - \\mu_{1} \\mid$. The means and variances of the two representative groups are $(\\mu_1,\\sigma_{1}^2)$ and $(\\mu_2,\\sigma_{2}^2)$.\n",
"</details>"
]
},
{
"cell_type": "markdown",
"id": "41271826",
"metadata": {},
"source": [
"Since the app has 1038 daily active users you will need to determine for how long should you run the experiment to get the desired number of users. Assuming you will split your users 50-50 between the original app and the version with the feature you would have:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "44e5c6a3",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"AB test should run for 9 days to gather enough data\n"
]
}
],
"source": [
"daily_active_users = 1038\n",
"\n",
"n_days = math.ceil((required_sample_size*2)/daily_active_users)\n",
"\n",
"print(f\"AB test should run for {n_days} days to gather enough data\")"
]
},
{
"cell_type": "markdown",
"id": "8b3f852e",
"metadata": {},
"source": [
"This is a very important step in AB testing because you want to have a big enough sample size so you can trust the results but you don't want to run the experiment forever because this increases the chances of any external factor messing up the effect of the feature you want to capture. Also you don't know if the new feature will even be beneficial so keeping the experiment short minimizes the risk of damaging the overall conversion rate. Run the experiment by running the cell below:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "2ec85093",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>user_id</th>\n",
" <th>user_type</th>\n",
" <th>converted</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>PXDWVA9HRZ</td>\n",
" <td>variation</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>JUM5M79PT6</td>\n",
" <td>control</td>\n",
" <td>1</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>R1B0UGE669</td>\n",
" <td>variation</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>48V3YQJ6WI</td>\n",
" <td>control</td>\n",
" <td>0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>GDEOU9HA2P</td>\n",
" <td>control</td>\n",
" <td>0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" user_id user_type converted\n",
"0 PXDWVA9HRZ variation 0\n",
"1 JUM5M79PT6 control 1\n",
"2 R1B0UGE669 variation 0\n",
"3 48V3YQJ6WI control 0\n",
"4 GDEOU9HA2P control 0"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data = utils.run_ab_test_personalized_feed(n_days)\n",
"\n",
"data.head(5)"
]
},
{
"cell_type": "markdown",
"id": "b5453e4c",
"metadata": {},
"source": [
"Similarly to the data in section 1, you have the information of the type of group and whether or not the user converted, for every user. Separate the two groups by running the next cell:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "97c64b4a",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"4632 users saw the original app with an average CVR of 0.1244\n",
"\n",
"4728 users saw the app with the new feature with an average CVR of 0.1415\n"
]
}
],
"source": [
"control_data = data[data[\"user_type\"]==\"control\"][\"converted\"]\n",
"variation_data = data[data[\"user_type\"]==\"variation\"][\"converted\"]\n",
"\n",
"print(f\"{len(control_data)} users saw the original app with an average CVR of {control_data.mean():.4f}\\n\")\n",
"print(f\"{len(variation_data)} users saw the app with the new feature with an average CVR of {variation_data.mean():.4f}\")"
]
},
{
"cell_type": "markdown",
"id": "84ae93c8",
"metadata": {},
"source": [
"The split is not perfectly balanced but you have enough data for each group to reach a conclusion.\n",
"\n",
"At first glance it looks like the new feature did in fact improve the user experience and drived more users to convert. However you already know you must perform a hypothesis test to know if there is a significant difference between the **rates (proportions)** of these two segments. You can do this by computing the z-statistic:\n",
"\n",
"$$ z = \\frac{\\hat{p}_1 - \\hat{p}_2}{\\sqrt{\\hat{p}(1-\\hat{p})\\left(\\frac{1}{n_1} + \\frac{1}{n_2}\\right)}}$$\n",
"\n",
"where $\\hat{p}$ is the pooled proportion: $\\hat{p} = \\frac{x_1 + x_2}{n_1 + n_2}$\n",
"\n",
"The next step is to compute all the necessary metrics for every group. For this you decide to use a dataclass that holds this information:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "6efae0b9",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"@dataclass\n",
"class estimation_metrics_prop:\n",
" n: int\n",
" x: int\n",
" p: float\n",
" \n",
" def __repr__(self):\n",
" return f\"sample_params(n={self.n}, x={self.x}, p={self.p:.3f})\""
]
},
{
"cell_type": "markdown",
"id": "0397ed43",
"metadata": {},
"source": [
"This class will hold the information for $n$, $x$ and $p$. \n",
"\n",
"## Exercise 5: compute_proportion_metrics\n",
"\n",
"Now that you have a container for all these metrics it is your job to code a function that given some data will compute them for that particular set of data. To do this complete the `compute_proportion_metrics` below.\n",
"\n",
"Hints:\n",
"- $n$ stands for the number of users in the data\n",
"- $x$ stands for the number of users who converted in the data\n",
"- $p$ stands for CVR (users who converted/total users\n",
"- `compute_proportion_metrics` expects a Pandas series as a parameter. You can sum all the values of a Pandas series by using the `sum()` method like this: `series.sum()`."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "75e8306e",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def compute_proportion_metrics(data):\n",
" \"\"\"Computes the relevant metrics out of a sample for proportion-like data.\n",
"\n",
" Args:\n",
" data (pandas.core.series.Series): The sample data. In this case 1 if the user converted and 0 otherwise.\n",
"\n",
" Returns:\n",
" estimation_metrics_prop: The metrics saved in a dataclass instance.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" metrics = estimation_metrics_prop( \n",
" n=len(data),\n",
" x=data.sum(),\n",
" p=data.mean()\n",
" )\n",
" ### END CODE HERE ###\n",
" \n",
" return metrics"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "7b03f33a",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"n=4, x=2 and p=0.5000 for sample array\n",
"\n",
"n=4632, x=576 and p=0.1244 for control data\n",
"\n",
"n=4728, x=669 and p=0.1415 for variation data\n"
]
}
],
"source": [
"# Test your code\n",
"cm = compute_proportion_metrics(np.array([1,0,0,1]))\n",
"print(f\"n={cm.n}, x={cm.x} and p={cm.p:.4f} for sample array\\n\")\n",
"\n",
"control_metrics = compute_proportion_metrics(control_data)\n",
"print(f\"n={control_metrics.n}, x={control_metrics.x} and p={control_metrics.p:.4f} for control data\\n\")\n",
"\n",
"variation_metrics = compute_proportion_metrics(variation_data)\n",
"print(f\"n={variation_metrics.n}, x={variation_metrics.x} and p={variation_metrics.p:.4f} for variation data\")"
]
},
{
"cell_type": "markdown",
"id": "57f288ac",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"n=4, x=2 and p=0.5000 for sample array\n",
"\n",
"n=4632, x=576 and p=0.1244 for control data\n",
"\n",
"n=4728, x=669 and p=0.1415 for variation data\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "a7f0401d",
"metadata": {},
"source": [
"## Exercise 6: pooled_proportion\n",
"\n",
"Now that you have a way of computing all necessary metrics for each sample it is time to create a way to compute the pooled proportion. For this fill the `pooled_proportion` function below. Notice that this function will receive two instances of the `estimation_metrics_prop` class.\n",
"\n",
"Remember that the pooled proportion can be computed like this: \n",
"\n",
"$\\hat{p} = \\frac{x_1 + x_2}{n_1 + n_2}$"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "12c6c01f",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def pooled_proportion(control_metrics, variation_metrics):\n",
" \"\"\"Compute the pooled proportion for the two samples.\n",
"\n",
" Args:\n",
" control_metrics (estimation_metrics_prop): The metrics for the control sample.\n",
" variation_metrics (estimation_metrics_prop): The metrics for the variation sample.\n",
"\n",
" Returns:\n",
" numpy.float: The pooled proportion.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" \n",
" x1, n1 = control_metrics.x, control_metrics.n\n",
" x2, n2 = variation_metrics.x, variation_metrics.n\n",
" \n",
" pp = (x1+x2)/(n1+n2)\n",
" \n",
" ### END CODE HERE ###\n",
" \n",
" return pp"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "360bafe1",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"pooled proportion for example arrays: 0.7143\n",
"\n",
"pooled proportion for AB test samples: 0.1330\n"
]
}
],
"source": [
"# Test your code\n",
"\n",
"test_m1, test_m2 = compute_proportion_metrics(np.array([1,0,1])), compute_proportion_metrics(np.array([1,1,1,0]))\n",
"pp = pooled_proportion(test_m1, test_m2)\n",
"print(f\"pooled proportion for example arrays: {pp:.4f}\\n\")\n",
"\n",
"pp = pooled_proportion(control_metrics, variation_metrics)\n",
"print(f\"pooled proportion for AB test samples: {pp:.4f}\")"
]
},
{
"cell_type": "markdown",
"id": "34b0919c",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"pooled proportion for example arrays: 0.7143\n",
"\n",
"pooled proportion for AB test samples: 0.1330\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "881d5a50",
"metadata": {},
"source": [
"## Exercise 7: z_statistic_diff_proportions\n",
"\n",
"Now you have everything you need to calculate the z-statistic for the difference between proportions. Remember that this statistic can be computed like this:\n",
"\n",
"$$ z = \\frac{\\hat{p}_1 - \\hat{p}_2}{\\sqrt{\\hat{p}(1-\\hat{p})\\left(\\frac{1}{n_1} + \\frac{1}{n_2}\\right)}}$$\n",
"\n",
"where $\\hat{p}$ is the pooled proportion: $\\hat{p} = \\frac{x_1 + x_2}{n_1 + n_2}$\n",
"\n",
"Hints:\n",
"- Remember to use the `pooled_proportion` function you coded earlier."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "96f0af5f",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def z_statistic_diff_proportions(control_metrics, variation_metrics):\n",
" \"\"\"Compute the z-statistic for the difference of two proportions.\n",
"\n",
" Args:\n",
" control_metrics (estimation_metrics_prop): The metrics for the control sample.\n",
" variation_metrics (estimation_metrics_prop): The metrics for the variation sample.\n",
"\n",
" Returns:\n",
" numpy.float: The z-statistic.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" \n",
" pp = pooled_proportion(control_metrics, variation_metrics)\n",
" \n",
" n1, p1 = control_metrics.n, control_metrics.p\n",
" n2, p2 = variation_metrics.n, variation_metrics.p\n",
" \n",
" z = (p1 - p2) / np.sqrt(pp * (1 - pp) * ((1 / n1) + (1 / n2)))\n",
" \n",
" ### END CODE HERE ###\n",
" \n",
" \n",
" return z"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "7fbe9143",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"z statistic for example arrays: -0.2415\n",
"\n",
"z statistic for AB test: -2.4422\n"
]
}
],
"source": [
"# Test your code\n",
"\n",
"z = z_statistic_diff_proportions(test_m1, test_m2)\n",
"print(f\"z statistic for example arrays: {z:.4f}\\n\")\n",
"\n",
"z = z_statistic_diff_proportions(control_metrics, variation_metrics)\n",
"print(f\"z statistic for AB test: {z:.4f}\")"
]
},
{
"cell_type": "markdown",
"id": "5ac834bc",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"z statistic for example arrays: -0.2415\n",
"\n",
"z statistic for AB test: -2.4422\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "63cd2ddf",
"metadata": {},
"source": [
"## Exercise 8: reject_nh_z_statistic\n",
"\n",
"Complete the `reject_nh_z_statistic` function below. This function should return whether to reject (or not) the null hypothesis by using the p-value method given the value of the z-statistic and a significance level. **This should be a two-sided test.**\n",
"\n",
"\n",
"Hints:\n",
"- You can use the `cdf` method from the [stats.norm](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) class to compute the p-value, as you did in exercise 4. **Don't forget to multiply your result by 2 since this is a two-sided test.**\n",
"\n",
"<!-- --> \n",
"\n",
"\n",
"- When passing the value of the z-statistic to the `cdf` function, you should provide the absolute value. You can achieve this by using Python's built-in `abs` function.\n",
"\n",
"<!-- --> \n",
"\n",
"\n",
"- If the p-value is lower than `alpha` then you should reject the null hypothesis."
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "ce9b0b2a",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def reject_nh_z_statistic(z_statistic, alpha=0.05):\n",
" \"\"\"Decide whether to reject (or not) the null hypothesis of the z-test.\n",
"\n",
" Args:\n",
" z_statistic (numpy.float): The computed value of the z-statistic for the two proportions.\n",
" alpha (float, optional): The desired significance level. Defaults to 0.05.\n",
"\n",
" Returns:\n",
" bool: True if the null hypothesis should be rejected. False otherwise.\n",
" \"\"\"\n",
" \n",
" reject = False\n",
" ### START CODE HERE ###\n",
" p_value = 2 * (1 - stats.norm.cdf(np.abs(z_statistic)))\n",
" \n",
" if p_value < alpha:\n",
" reject = True\n",
" ### END CODE HERE ###\n",
" \n",
" return reject"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "6402562c",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The null hypothesis can be rejected at the 0.05 significance level: True\n",
"\n",
"There is enough statistical evidence against H0.\n",
"Thus it can be concluded that there is a statistically significant difference between the two proportions.\n"
]
}
],
"source": [
"# Test your code\n",
"\n",
"alpha = 0.05\n",
"reject_nh = reject_nh_z_statistic(z, alpha)\n",
"\n",
"print(f\"The null hypothesis can be rejected at the {alpha} significance level: {reject_nh}\\n\")\n",
"\n",
"msg = \"\" if reject_nh else \" not\"\n",
"print(f\"There is{msg} enough statistical evidence against H0.\\nThus it can be concluded that there is{msg} a statistically significant difference between the two proportions.\")"
]
},
{
"cell_type": "markdown",
"id": "ed91c114",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"The null hypothesis can be rejected at the 0.05 significance level: True\n",
"\n",
"There is enough statistical evidence against H0.\n",
"Thus it can be concluded that there is a statistically significant difference between the two proportions\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "2d38d2e5",
"metadata": {},
"source": [
"In this case the new feature did in fact increased the CVR. The conclusion of the AB test is that you should release the new feature to all users as there is strong statistical evidence that this will result in a better CVR."
]
},
{
"cell_type": "markdown",
"id": "acdeaba3",
"metadata": {},
"source": [
"## Exercise_9: confidence_interval_proportion\n",
"\n",
"Finally you would like to create confidence intervals for the CVRs of each one of the two groups. You can compute such interval for a proportion like this:\n",
"\n",
"$$ \\hat{p} \\pm z_{1-\\alpha/2} \\sqrt{\\frac{\\hat{p}(1-\\hat{p})}{n}}$$\n",
"\n",
"Complete the `confidence_interval_proportion` function below. This function will receive the metrics of one of the groups and should return the lower and upper values of the confidence interval.\n",
"\n",
"Hints:\n",
"- You can use the `ppf` method from the [stats.norm](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) class to compute the value of $z$"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "e1f31cec",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [],
"source": [
"def confidence_interval_proportion(metrics, alpha=0.05):\n",
" \"\"\"Compute the confidende interval for a proportion-like sample.\n",
"\n",
" Args:\n",
" metrics (estimation_metrics_prop): The metrics for the sample.\n",
" alpha (float, optional): The desired significance level. Defaults to 0.05.\n",
"\n",
" Returns:\n",
" (numpy.float, numpy.float): The lower and upper bounds of the confidence interval.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" n, p = metrics.n, metrics.p\n",
" \n",
" distance = stats.norm.ppf(1 - alpha / 2) * np.sqrt(p * (1 - p) / n)\n",
" \n",
" lower = p - distance\n",
" upper = p + distance\n",
" ### END CODE HERE ###\n",
" \n",
" return lower, upper"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "d1d26bbf",
"metadata": {
"tags": [
"graded"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Confidence interval for control group: [0.115, 0.134]\n",
"\n",
"Confidence interval for variation group: [0.132, 0.151]\n"
]
}
],
"source": [
"# Test your code\n",
"\n",
"c_lower, c_upper = confidence_interval_proportion(control_metrics)\n",
"print(f\"Confidence interval for control group: [{c_lower:.3f}, {c_upper:.3f}]\\n\")\n",
"\n",
"v_lower, v_upper = confidence_interval_proportion(variation_metrics)\n",
"print(f\"Confidence interval for variation group: [{v_lower:.3f}, {v_upper:.3f}]\")"
]
},
{
"cell_type": "markdown",
"id": "dc13cb67",
"metadata": {},
"source": [
"##### __Expected Output__\n",
"\n",
"```\n",
"Confidence interval for control group: [0.115, 0.134]\n",
"\n",
"Confidence interval for variation group: [0.132, 0.151]\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "9dc57930",
"metadata": {},
"source": [
"As you can see the intervals for the two groups do not overlap, which alligns with the conclusion that you found earlier that there is indeed a statistically significant difference between the two proportions."
]
},
{
"cell_type": "markdown",
"id": "ef858819",
"metadata": {},
"source": [
"## Bonus Widget: AB test calculator\n",
"\n",
"If you use any web search engine you will find a lot of AB test calculators online but they usually just provide a result with no real explanation of how these computations are made. After finishing this assignment you know what is going on behind the scenes so you decide to create your own AB test calculator for future uses. This can be accomplished by using some python widgets and the functions you just coded.\n",
"\n",
"Run the next cell to render the calculator with your functions (`z_statistic_diff_proportions` and `reject_nh_z_statistic`) as backend:"
]
},
{
"cell_type": "code",
"execution_count": 33,
"id": "8a8e78aa",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "54446d55b5cd4d04aac94791e718e795",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"interactive(children=(IntText(value=4632, description='Users A:'), IntText(value=576, description='Conversions…"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"utils.AB_test_dashboard(z_statistic_diff_proportions, reject_nh_z_statistic)"
]
},
{
"cell_type": "markdown",
"id": "13632472",
"metadata": {},
"source": [
"**Congratulations on finishing this assignment!**\n",
"\n",
"Now you have created all the required steps to perform an AB test for continuous and proportion-based metrics. \n",
"\n",
"**This is the last assignment of the course and the specialization so give yourself a pat on the back for such a great accomplishment! Nice job!!!!**"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "33d081c8",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"grader_version": "1",
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}