Mercurial > hg > speedtest

annotate README.md @ 0:26e919a36f86 default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
speedtest containerized dispatching software
author Jeff Hammel <k0scist@gmail.com>
date Thu, 09 Feb 2017 09:47:04 -0800
parents
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
1 # Speedtest
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
2
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
3 ## The Pattern
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
4
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
5 `speedtest`
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
6 will use this pattern for outer loop testing:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
7
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
8 - a [Docker](http://docker.io/) container is created from a
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
9 `speedtest` script,
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
10 and a docker
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
11 [CMD]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
12 (https://docs.docker.com/v1.8/reference/builder/#cmd)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
13 that will run
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
14 the appropriate test and post a JSON blob to an
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
15 [Elasticsearch](https://www.elastic.co/) node
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
16
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
17 - the Docker container is dispatched to a container service
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
18 such that it may be run at
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
19 cloud scale and from a variety of regions
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
20
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
21 - the test [CMD]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
22 (http://docs.docker.com/engine/reference/builder/#cmd)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
23 of the container contains steps that will generate
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
24 JSON data and post this to an Elasticsearch node. This
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
25 Elasticsearch data may then be analyzed and utilized for reporting
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
26 of system status.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
27
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
28 This pattern is of general utility for blackbox testing and monitoring
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
29 of a web system under test. Monitoring may be performed in a continuous
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
30 and ongoing basis, the Elasticsearch data combed for performance information.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
31
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
32
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
33 ## Installation
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
34
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
35 `speedtest` follows fairly standard
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
36 [python packaging]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
37 (https://python-packaging-user-guide.readthedocs.org/en/latest/)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
38 practices. We do need other software, like
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
39 [docker](http://docker.io), to help us along.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
40
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
41 Let's get started!
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
42
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
43
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
44 ## What Speedtest Gives You
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
45
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
46 `speedtest` attempts to provide a suite of tools for making worldwide
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
47 dockerized speedtests a reality. Many of these are provide in terms
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
48 of [setuptools](https://pythonhosted.org/setuptools/)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
49 [entry points]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
50 (https://pythonhosted.org/setuptools/setuptools.html#dynamic-discovery-of-services-and-plugins),
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
51 including
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
52 [console scripts]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
53 (https://pythonhosted.org/setuptools/setuptools.html#automatic-script-creation).
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
54
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
55 Several console scripts are also installed with `speedtest`.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
56 All provided console scripts utilize
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
57 [argparse](https://docs.python.org/2/library/argparse.html)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
58 for their command line interface. Detailed usage information for a
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
59 script may be obtained by using the `--help` command line switch.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
60 For example: `speedtest --help` .
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
61
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
62
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
63 ## About Speedtest
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
64
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
65 `speedtest` is a collection of scripts and utilities to make real the
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
66 pattern of docker container dispatch for data
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
67 gathering via JSON output to Elasticsearch.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
68
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
69 While `Dockerfile`s are a good abstraction layer for building
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
70 [linux containers]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
71 (https://en.wikipedia.org/wiki/LXC),
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
72 often the run steps must be automated and parameterized.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
73
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
74 ## speedtest with local results
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
75
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
76 Usage information is available with the `--help` flag:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
77
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
78 speedtest --help
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
79
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
80 Arbitrary metadata may be added with the `--variables` flag. While the results processing and information available should be refined, right now we are erring on the side of too much, which is at least fairly flexible.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
81
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
82 To run a speedtest versus the URL http://k0s.org/playground/1M.bin adding the variable `foo` with value of `bar` with local JSON results measuring every 100k, do:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
83
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
84 speedtest --variables foo=bar --format json --chunk $((100*1024)) http://k0s.org/playground/1M.bin
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
85
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
86
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
87 ## Containers
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
88
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
89 ### Containerized Download Speed Testing
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
90
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
91 `speedtest.py`
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
92
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
93 (and, equivalently, the `speedtest` [console script]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
94 (http://python-packaging.readthedocs.org/en/latest/command-line-scripts.html#the-console-scripts-entry-point))
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
95 measures download speed over time given a URL:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
96
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
97 # speedtest http://k0s.org/playground/1M.bin
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
98 ...
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
99 {"duration": 1.7867929935455322, "cumulative": 586848, "speed": 209367, "size": 1048576}
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
100
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
101
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
102 ### Containerized Dispatching
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
103
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
104 - pull the Docker image from a container registry
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
105 - run the Docker image
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
106
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
107 ### Container Registry
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
108
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
109 In order to make use of containerized dispatch, you may want a
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
110 [docker registry](https://docs.docker.com/registry/):
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
111
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
112 - https://docs.docker.com/registry/deploying/
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
113 - https://www.docker.com/docker-trusted-registry
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
114 - https://github.com/docker/distribution
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
115 - https://www.docker.com/aws
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
116
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
117 [Docker hub](https://hub.docker.com/)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
118 has an official [Docker](https://www.docker.com)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
119 repository for a
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
120 [Docker registry]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
121 (https://hub.docker.com/_/registry/). How meta is that? You can get
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
122 started just by running:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
123
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
124 docker run registry:2
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
125
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
126
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
127 ## Elasticsearch
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
128
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
129 _"Elasticsearch is a search server based on Lucene. It provides a
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
130 distributed, multitenant-capable full-text search engine with a HTTP
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
131 web interface and schema-free JSON documents. Elasticsearch is
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
132 developed in Java and is released as open source under the terms of
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
133 the Apache License."_,
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
134 [Wikipedia](https://en.wikipedia.org/wiki/Elasticsearch)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
135
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
136 The schema-free data store together with the easy RESTful interface
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
137 make [Elasticsearch]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
138 (https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
139 a great place for `speedtest` to put its results.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
140
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
141 There is a docker container for Elasticsearch available:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
142 https://hub.docker.com/_/elasticsearch/
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
143
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
144 The results of `speedtest` may be posted to Elasticsearch:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
145
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
146 # curl -XPOST localhost:9200/speedtest/results?pretty -d "$(speedtest http://k0s.org/playground/1M.bin | tail -n 1)"
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
147 {
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
148 "_index" : "speedtest",
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
149 "_type" : "results",
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
150 "_id" : "AVGNmWMTVkaacKUEikLl",
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
151 "_version" : 1,
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
152 "_shards" : {
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
153 "total" : 2,
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
154 "successful" : 1,
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
155 "failed" : 0
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
156 },
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
157 "created" : true
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
158 }
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
159
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
160 However, for lots of data we probably want to use the
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
161 [bulk API]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
162 (https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
163
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
164 # speedtest 'http://k0s.org/playground/1M.bin' --fmt elasticsearch --chunk $((1024*1024)) -o 'http://1.2.3.4:32793/speedtest/results/_bulk'
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
165 {"took":1,"errors":false,"items":[{"create":{"_index":"speedtest","_type":"results","_id":"AVVU7PIeZtDhCquHQrZV","_version":1,"status":201}}]}
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
166
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
167 Note that the index and type *must* be specified in the output URL for this to work.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
168 See https://www.elastic.co/blog/index-vs-type for clarification of why this is a
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
169 good thing.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
170
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
171
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
172 JSON blobs are uploaded to an Elasticsearch instance.
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
173
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
174
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
175 ## Links
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
176
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
177 , ,
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
178 |`-._ _.-'|
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
179 \ `-' /
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
180 `-._ _.-'
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
181 `#'
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
182 ~~~~~~~~~~~~~
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
183
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
184 [Docker](http://docker.com/):
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
185
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
186 - https://docs.docker.com/machine/get-started/
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
187 - https://github.com/victorlin/crane
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
188 - https://docker-py.readthedocs.org/en/latest/
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
189
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
190
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
191 Elasticsearch:
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
192 - [The Definitive Guide](https://www.elastic.co/guide/en/elasticsearch/guide/index.html)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
193 - https://www.elastic.co/guide/en/elasticsearch/reference/current/_modifying_your_data.html
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
194 - https://hub.docker.com/_/elasticsearch/
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
195 - [Python Elasticsearch Client]
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
196 (http://elasticsearch-py.readthedocs.org/en/master/index.html)
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
197 - https://bitquabit.com/post/having-fun-python-and-elasticsearch-part-1/
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
198
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
199
26e919a36f86 speedtest containerized dispatching software
Jeff Hammel <k0scist@gmail.com>
parents:
diff changeset
200 [Jeff Hammel](https://hub.docker.com/u/jehammel/)