ipynb • Lines: 533{
"cells": [
{
"cell_type": "markdown",
"id": "reverse-interview",
"metadata": {},
"source": [
"# Vector Operations: Scalar Multiplication, Sum and Dot Product of Vectors"
]
},
{
"cell_type": "markdown",
"id": "parental-conclusion",
"metadata": {},
"source": [
"In this lab you will use Python and `NumPy` functions to perform main vector operations: scalar multiplication, sum of vectors and their dot product. You will also investigate the speed of calculations using loop and vectorized forms of these main linear algebra operations"
]
},
{
"cell_type": "markdown",
"id": "looking-barcelona",
"metadata": {},
"source": [
"# Table of Contents\n",
"- [ 1 - Scalar Multiplication and Sum of Vectors](#1)\n",
" - [ 1.1 - Visualization of a Vector $v\\in\\mathbb{R}^2$](#1.1)\n",
" - [ 1.2 - Scalar Multiplication](#1.2)\n",
" - [ 1.3 - Sum of Vectors](#1.3)\n",
" - [ 1.4 - Norm of a Vector](#1.4)\n",
"- [ 2 - Dot Product](#2)\n",
" - [ 2.1 - Algebraic Definition of the Dot Product](#2.1)\n",
" - [ 2.2 - Dot Product using Python](#2.2)\n",
" - [ 2.3 - Speed of Calculations in Vectorized Form](#2.3)\n",
" - [ 2.4 - Geometric Definition of the Dot Product](#2.4)\n",
" - [ 2.5 - Application of the Dot Product: Vector Similarity](#2.5)"
]
},
{
"cell_type": "markdown",
"id": "advance-butler",
"metadata": {},
"source": [
"## Packages\n",
"\n",
"Load the `NumPy` package to access its functions."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "promotional-buffer",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "markdown",
"id": "severe-studio",
"metadata": {},
"source": [
"<a name='1'></a>\n",
"## 1 - Scalar Multiplication and Sum of Vectors"
]
},
{
"cell_type": "markdown",
"id": "ethical-success",
"metadata": {},
"source": [
"<a name='1.1'></a>\n",
"### 1.1 - Visualization of a Vector $v\\in\\mathbb{R}^2$\n",
"\n",
"You already have seen in the videos and labs, that vectors can be visualized as arrows, and it is easy to do it for a $v\\in\\mathbb{R}^2$, e.g.\n",
"$v=\\begin{bmatrix}\n",
" 1 & 3\n",
"\\end{bmatrix}^T$\n",
"\n",
"The following code will show the visualization."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "korean-landing",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_vectors(list_v, list_label, list_color):\n",
" _, ax = plt.subplots(figsize=(10, 10))\n",
" ax.tick_params(axis='x', labelsize=14)\n",
" ax.tick_params(axis='y', labelsize=14)\n",
" ax.set_xticks(np.arange(-10, 10))\n",
" ax.set_yticks(np.arange(-10, 10))\n",
" \n",
" \n",
" plt.axis([-10, 10, -10, 10])\n",
" for i, v in enumerate(list_v):\n",
" sgn = 0.4 * np.array([[1] if i==0 else [i] for i in np.sign(v)])\n",
" plt.quiver(v[0], v[1], color=list_color[i], angles='xy', scale_units='xy', scale=1)\n",
" ax.text(v[0]-0.2+sgn[0], v[1]-0.2+sgn[1], list_label[i], fontsize=14, color=list_color[i])\n",
"\n",
" plt.grid()\n",
" plt.gca().set_aspect(\"equal\")\n",
" plt.show()\n",
"\n",
"v = np.array([[1],[3]])\n",
"# Arguments: list of vectors as NumPy arrays, labels, colors.\n",
"plot_vectors([v], [f\"$v$\"], [\"black\"])"
]
},
{
"cell_type": "markdown",
"id": "original-translator",
"metadata": {},
"source": [
"The vector is defined by its **norm (length, magnitude)** and **direction**, not its actual position. But for clarity and convenience vectors are often plotted starting in the origin (in $\\mathbb{R}^2$ it is a point $(0,0)$) ."
]
},
{
"cell_type": "markdown",
"id": "speaking-surgeon",
"metadata": {},
"source": [
"<a name='1.2'></a>\n",
"### 1.2 - Scalar Multiplication\n",
"\n",
"**Scalar multiplication** of a vector $v=\\begin{bmatrix}\n",
" v_1 & v_2 & \\ldots & v_n \n",
"\\end{bmatrix}^T\\in\\mathbb{R}^n$ by a scalar $k$ is a vector $kv=\\begin{bmatrix}\n",
" kv_1 & kv_2 & \\ldots & kv_n \n",
"\\end{bmatrix}^T$ (element by element multiplication). If $k>0$, then $kv$ is a vector pointing in the same direction as $v$ and it is $k$ times as long as $v$. If $k=0$, then $kv$ is a zero vector. If $k<0$, vector $kv$ will be pointing in the opposite direction. In Python you can perform this operation with a `*` operator. Check out the example below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "acute-investment",
"metadata": {},
"outputs": [],
"source": [
"plot_vectors([v, 2*v, -2*v], [f\"$v$\", f\"$2v$\", f\"$-2v$\"], [\"black\", \"green\", \"blue\"])"
]
},
{
"cell_type": "markdown",
"id": "civil-county",
"metadata": {},
"source": [
"<a name='1.3'></a>\n",
"### 1.3 - Sum of Vectors\n",
"\n",
"**Sum of vectors (vector addition)** can be performed by adding the corresponding components of the vectors: if $v=\\begin{bmatrix}\n",
" v_1 & v_2 & \\ldots & v_n \n",
"\\end{bmatrix}^T\\in\\mathbb{R}^n$ and \n",
"$w=\\begin{bmatrix}\n",
" w_1 & w_2 & \\ldots & w_n \n",
"\\end{bmatrix}^T\\in\\mathbb{R}^n$, then $v + w=\\begin{bmatrix}\n",
" v_1 + w_1 & v_2 + w_2 & \\ldots & v_n + w_n \n",
"\\end{bmatrix}^T\\in\\mathbb{R}^n$. The so-called **parallelogram law** gives the rule for vector addition. For two vectors $u$ and $v$ represented by the adjacent sides (both in magnitude and direction) of a parallelogram drawn from a point, the vector sum $u+v$ is is represented by the diagonal of the parallelogram drawn from the same point:\n",
"\n",
"<img src = \"images/sum_of_vectors.png\" width=\"230\" align=\"middle\"/>\n",
"\n",
"In Python you can either use `+` operator or `NumPy` function `np.add()`. In the following code you can uncomment the line to check that the result will be the same:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "acoustic-heath",
"metadata": {},
"outputs": [],
"source": [
"v = np.array([[1],[3]])\n",
"w = np.array([[4],[-1]])\n",
"\n",
"plot_vectors([v, w, v + w], [f\"$v$\", f\"$w$\", f\"$v + w$\"], [\"black\", \"black\", \"red\"])\n",
"# plot_vectors([v, w, np.add(v, w)], [f\"$v$\", f\"$w$\", f\"$v + w$\"], [\"black\", \"black\", \"red\"])"
]
},
{
"cell_type": "markdown",
"id": "nearby-portal",
"metadata": {},
"source": [
"<a name='1.4'></a>\n",
"### 1.4 - Norm of a Vector\n",
"\n",
"The norm of a vector $v$ is denoted as $\\lvert v\\rvert$. It is a nonnegative number that describes the extent of the vector in space (its length). The norm of a vector can be found using `NumPy` function `np.linalg.norm()`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "spare-timing",
"metadata": {},
"outputs": [],
"source": [
"print(\"Norm of a vector v is\", np.linalg.norm(v))"
]
},
{
"cell_type": "markdown",
"id": "talented-survey",
"metadata": {},
"source": [
"<a name='2'></a>\n",
"## 2 - Dot Product"
]
},
{
"cell_type": "markdown",
"id": "sunset-transmission",
"metadata": {},
"source": [
"<a name='2.1'></a>\n",
"### 2.1 - Algebraic Definition of the Dot Product\n",
"\n",
"The **dot product** (or **scalar product**) is an algebraic operation that takes two vectors $x=\\begin{bmatrix}\n",
" x_1 & x_2 & \\ldots & x_n \n",
"\\end{bmatrix}^T\\in\\mathbb{R}^n$ and \n",
"$y=\\begin{bmatrix}\n",
" y_1 & y_2 & \\ldots & y_n \n",
"\\end{bmatrix}^T\\in\\mathbb{R}^n$ and returns a single scalar. The dot product can be represented with a dot operator $x\\cdot y$ and defined as:\n",
"\n",
"$$x\\cdot y = \\sum_{i=1}^{n} x_iy_i = x_1y_1+x_2y_2+\\ldots+x_ny_n \\tag{1}$$"
]
},
{
"cell_type": "markdown",
"id": "meaningful-timer",
"metadata": {},
"source": [
"<a name='2.2'></a>\n",
"### 2.2 - Dot Product using Python\n",
"\n",
"The simplest way to calculate dot product in Python is to take the sum of element by element multiplications. You can define the vectors $x$ and $y$ by listing their coordinates:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "musical-battlefield",
"metadata": {},
"outputs": [],
"source": [
"x = [1, -2, -5]\n",
"y = [4, 3, -1]"
]
},
{
"cell_type": "markdown",
"id": "plastic-temple",
"metadata": {},
"source": [
"Next, let’s define a function `dot(x,y)` for the dot product calculation:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "signed-syndicate",
"metadata": {},
"outputs": [],
"source": [
"def dot(x, y):\n",
" s=0\n",
" for xi, yi in zip(x, y):\n",
" s += xi * yi\n",
" return s"
]
},
{
"cell_type": "markdown",
"id": "upper-highlight",
"metadata": {},
"source": [
"For the sake of simplicity, let’s assume that the vectors passed to the above function are always of the same size, so that you don’t need to perform additional checks.\n",
"\n",
"Now everything is ready to perform the dot product calculation calling the function `dot(x,y)`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "amazing-broadway",
"metadata": {},
"outputs": [],
"source": [
"print(\"The dot product of x and y is\", dot(x, y))"
]
},
{
"cell_type": "markdown",
"id": "banned-dallas",
"metadata": {},
"source": [
"Dot product is very a commonly used operator, so `NumPy` linear algebra package provides quick way to calculate it using function `np.dot()`:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "accessible-kinase",
"metadata": {},
"outputs": [],
"source": [
"print(\"np.dot(x,y) function returns dot product of x and y:\", np.dot(x, y)) "
]
},
{
"cell_type": "markdown",
"id": "friendly-beast",
"metadata": {},
"source": [
"Note that you did not have to define vectors $x$ and $y$ as `NumPy` arrays, the function worked even with the lists. But there are alternative functions in Python, such as explicit operator `@` for the dot product, which can be applied only to the `NumPy` arrays. You can run the following cell to check that."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "built-paper",
"metadata": {},
"outputs": [],
"source": [
"print(\"This line output is a dot product of x and y: \", np.array(x) @ np.array(y))\n",
"\n",
"print(\"\\nThis line output is an error:\")\n",
"try:\n",
" print(x @ y)\n",
"except TypeError as err:\n",
" print(err)"
]
},
{
"cell_type": "markdown",
"id": "central-museum",
"metadata": {},
"source": [
"As both `np.dot()` and `@` operators are commonly used, it is recommended to define vectors as `NumPy` arrays to avoid errors. Let's redefine vectors $x$ and $y$ as `NumPy` arrays to be safe:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "israeli-jumping",
"metadata": {},
"outputs": [],
"source": [
"x = np.array(x)\n",
"y = np.array(y)"
]
},
{
"cell_type": "markdown",
"id": "wicked-queensland",
"metadata": {},
"source": [
"<a name='2.3'></a>\n",
"### 2.3 - Speed of Calculations in Vectorized Form\n",
"\n",
"Dot product operations in Machine Learning applications are applied to the large vectors with hundreds or thousands of coordinates (called **high dimensional vectors**). Training models based on large datasets often takes hours and days even on powerful machines. Speed of calculations is crucial for the training and deployment of your models. \n",
"\n",
"It is important to understand the difference in the speed of calculations using vectorized and the loop forms of the vectors and functions. In the loop form operations are performed one by one, while in the vectorized form they can be performed in parallel. In the section above you defined loop version of the dot product calculation (function `dot()`), while `np.dot()` and `@` are the functions representing vectorized form.\n",
"\n",
"Let's perform a simple experiment to compare their speed. Define new vectors $a$ and $b$ of the same size $1,000,000$:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "amino-creation",
"metadata": {},
"outputs": [],
"source": [
"a = np.random.rand(1000000)\n",
"b = np.random.rand(1000000)"
]
},
{
"cell_type": "markdown",
"id": "facial-refrigerator",
"metadata": {},
"source": [
"Use `time.time()` function to evaluate amount of time (in seconds) required to calculate dot product using the function `dot(x,y)` which you defined above: "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "handed-influence",
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"\n",
"tic = time.time()\n",
"c = dot(a,b)\n",
"toc = time.time()\n",
"print(\"Dot product: \", c)\n",
"print (\"Time for the loop version:\" + str(1000*(toc-tic)) + \" ms\")"
]
},
{
"cell_type": "markdown",
"id": "accessible-sherman",
"metadata": {},
"source": [
"Now compare it with the speed of the vectorized versions:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "determined-cooking",
"metadata": {},
"outputs": [],
"source": [
"tic = time.time()\n",
"c = np.dot(a,b)\n",
"toc = time.time()\n",
"print(\"Dot product: \", c)\n",
"print (\"Time for the vectorized version, np.dot() function: \" + str(1000*(toc-tic)) + \" ms\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "scientific-empty",
"metadata": {},
"outputs": [],
"source": [
"tic = time.time()\n",
"c = a @ b\n",
"toc = time.time()\n",
"print(\"Dot product: \", c)\n",
"print (\"Time for the vectorized version, @ function: \" + str(1000*(toc-tic)) + \" ms\")"
]
},
{
"cell_type": "markdown",
"id": "useful-sleeping",
"metadata": {},
"source": [
"You can see that vectorization is extremely beneficial in terms of the speed of calculations!"
]
},
{
"cell_type": "markdown",
"id": "postal-latin",
"metadata": {},
"source": [
"<a name='2.4'></a>\n",
"### 2.4 - Geometric Definition of the Dot Product\n",
"\n",
"In [Euclidean space](https://en.wikipedia.org/wiki/Euclidean_space), a Euclidean vector has both magnitude and direction. The dot product of two vectors $x$ and $y$ is defined by:\n",
"\n",
"$$x\\cdot y = \\lvert x\\rvert \\lvert y\\rvert \\cos(\\theta),\\tag{2}$$\n",
"\n",
"where $\\theta$ is the angle between the two vectors:\n",
"\n",
"<img src = \"images/dot_product_geometric.png\" width=\"230\" align=\"middle\"/>\n",
"\n",
"This provides an easy way to test the orthogonality between vectors. If $x$ and $y$ are orthogonal (the angle between vectors is $90^{\\circ}$), then since $\\cos(90^{\\circ})=0$, it implies that **the dot product of any two orthogonal vectors must be $0$**. Let's test it, taking two vectors $i$ and $j$ we know are orthogonal:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "shared-climb",
"metadata": {},
"outputs": [],
"source": [
"i = np.array([1, 0, 0])\n",
"j = np.array([0, 1, 0])\n",
"print(\"The dot product of i and j is\", dot(i, j))"
]
},
{
"cell_type": "markdown",
"id": "thermal-railway",
"metadata": {},
"source": [
"<a name='2.5'></a>\n",
"### 2.5 - Application of the Dot Product: Vector Similarity\n",
"\n",
"Geometric definition of a dot product is used in one of the applications - to evaluate **vector similarity**. In Natural Language Processing (NLP) words or phrases from vocabulary are mapped to a corresponding vector of real numbers. Similarity between two vectors can be defined as a cosine of the angle between them. When they point in the same direction, their similarity is 1 and it decreases with the increase of the angle. \n",
"\n",
"Then equation $(2)$ can be rearranged to evaluate cosine of the angle between vectors:\n",
"\n",
"$\\cos(\\theta)=\\frac{x \\cdot y}{\\lvert x\\rvert \\lvert y\\rvert}.\\tag{3}$\n",
"\n",
"Zero value corresponds to the zero similarity between vectors (and words corresponding to those vectors). Largest value is when vectors point in the same direction, lowest value is when vectors point in the opposite directions.\n",
"\n",
"This example of vector similarity is given to link the material with the Machine Learning applications. There will be no actual implementation of it in this Course. Some examples of implementation can be found in the Natual Language Processing Specialization.\n",
"\n",
"Well done, you have finished this lab!"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "collect-needle",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
}
},
"nbformat": 4,
"nbformat_minor": 5
}