Page 1 of 2

lwip http server, can't read from socket

Posted: Sun Oct 30, 2016 5:29 pm
by engelant
So today I tried to create a litte http server, but I got stuck when reading from the accepted sockt. I don't know if this is directly esp related, but the lwip examples from the internet are supposed to work.

Code: Select all

/* Hello World Example

 This example code is in the Public Domain (or CC0 licensed, at your option.)

 Unless required by applicable law or agreed to in writing, this
 software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
 CONDITIONS OF ANY KIND, either express or implied.
 */
#include <stdio.h>
#include <string.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/event_groups.h"
#include "esp_system.h"
#include "esp_wifi.h"
#include "esp_event_loop.h"
#include "esp_log.h"
#include "nvs_flash.h"
#include "driver/ledc.h"
#include "lwip/err.h"
#include "lwip/arch.h"
#include "lwip/api.h"

/* FreeRTOS event group to signal when we are connected & ready to make a request */
static EventGroupHandle_t wifi_event_group;

/* The event group allows multiple bits for each event,
 but we only care about one event - are we connected
 to the AP with an IP? */
static const int CONNECTED_BIT = BIT0;

static const char *TAG = "example";


/* The examples use simple WiFi configuration that you can set via
 'make menuconfig'.

 If you'd rather not, just change the below entries to strings with
 the config you want - ie #define EXAMPLE_WIFI_SSID "mywifissid"
 */
#define EXAMPLE_WIFI_SSID "miau"
#define EXAMPLE_WIFI_PASS "**EDITED**"

static esp_err_t event_handler(void *ctx, system_event_t *event) {
	switch (event->event_id) {
	case SYSTEM_EVENT_STA_START:
		esp_wifi_connect();
		break;
	case SYSTEM_EVENT_STA_GOT_IP:
		xEventGroupSetBits(wifi_event_group, CONNECTED_BIT);
		break;
	case SYSTEM_EVENT_STA_DISCONNECTED:
		/* This is a workaround as ESP32 WiFi libs don't currently
		 auto-reassociate. */
		esp_wifi_connect();
		xEventGroupClearBits(wifi_event_group, CONNECTED_BIT);
		break;
	default:
		break;
	}
	return ESP_OK;
}

static void initialise_wifi(void) {
	tcpip_adapter_init();
	wifi_event_group = xEventGroupCreate();
	ESP_ERROR_CHECK(esp_event_loop_init(event_handler, NULL));
	wifi_init_config_t cfg = WIFI_INIT_CONFIG_DEFAULT()
	;
	ESP_ERROR_CHECK(esp_wifi_init(&cfg));
	ESP_ERROR_CHECK(esp_wifi_set_storage(WIFI_STORAGE_RAM));
	wifi_config_t wifi_config = { .sta = { .ssid = EXAMPLE_WIFI_SSID,
			.password = EXAMPLE_WIFI_PASS, }, };
	ESP_LOGI(TAG, "Setting WiFi configuration SSID %s...", wifi_config.sta.ssid);
	ESP_ERROR_CHECK(esp_wifi_set_mode(WIFI_MODE_STA));
	ESP_ERROR_CHECK(esp_wifi_set_config(WIFI_IF_STA, &wifi_config));
	ESP_ERROR_CHECK(esp_wifi_start());
}

#ifndef HTTPD_DEBUG
#define HTTPD_DEBUG         LWIP_DBG_OFF
#endif

const static char http_html_hdr[] =
		"HTTP/1.1 200 OK\r\nContent-type: text/html\r\n\r\n";
const static char http_index_html[] =
		"<html><head><title>Congrats!</title></head><body><h1>Welcome to our lwIP HTTP server!</h1><p>This is a small test page, served by httpserver-netconn.</body></html>";

const static char reply[] = "Ok, here you go";
/** Serve one HTTP connection accepted in the http thread */
static void http_server_netconn_serve(struct netconn *conn) {
	struct netbuf *inbuf;
	char *buf;
	uint16_t buflen;
	err_t err;

	/**
	 * THIS DOES NOT BELONG HERE
	 * It's just to make sure, the socket can be written to.
	 */
	netconn_write(conn, reply, sizeof(reply) - 1,
						NETCONN_NOCOPY);

	/* Read the data from the port, blocking if nothing yet there.
	 We assume the request (the part we care about) is in one netbuf */
	printf("About to read from socket\n");
	err = netconn_recv(conn, &inbuf);

	if (err == ERR_OK) {
		netbuf_data(inbuf, (void**) &buf, &buflen);

		/* Is this an HTTP GET command? (only check the first 5 chars, since
		 there are other formats for GET, and we're keeping it very simple )*/
		if (buflen >= 5 && buf[0] == 'G' && buf[1] == 'E' && buf[2] == 'T'
				&& buf[3] == ' ' && buf[4] == '/') {

			/* Send the HTML header
			 * subtract 1 from the size, since we dont send the \0 in the string
			 * NETCONN_NOCOPY: our data is const static, so no need to copy it
			 */
			netconn_write(conn, http_html_hdr, sizeof(http_html_hdr) - 1,
					NETCONN_NOCOPY);

			/* Send our HTML page */
			netconn_write(conn, http_index_html, sizeof(http_index_html) - 1,
					NETCONN_NOCOPY);
		}
	}
	/* Close the connection (server closes in HTTP) */
	netconn_close(conn);

	/* Delete the buffer (netconn_recv gives us ownership,
	 so we have to make sure to deallocate the buffer) */
	netbuf_delete(inbuf);
}

/** The main function, never returns! */
static void http_server_netconn_thread(void *arg) {
	struct netconn *conn, *newconn;
	err_t err;
	LWIP_UNUSED_ARG(arg);

	/* Create a new TCP connection handle */
	conn = netconn_new(NETCONN_TCP);
	LWIP_ERROR("http_server: invalid conn", (conn != NULL), return;);

	/* Bind to port 80 (HTTP) with default IP address */
	netconn_bind(conn, NULL, 80);

	/* Put the connection into LISTEN state */
	netconn_listen(conn);

	do {
		err = netconn_accept(conn, &newconn);
		if (err == ERR_OK) {
			printf("Connection accepted\n");
			http_server_netconn_serve(newconn);
			netconn_delete(newconn);
		}
	} while (err == ERR_OK); LWIP_DEBUGF(HTTPD_DEBUG,
			("http_server_netconn_thread: netconn_accept received error %d, shutting down",
					err));
	netconn_close(conn);
	netconn_delete(conn);
}

void app_main() {
	nvs_flash_init();
	system_init();
	initialise_wifi();
	//xTaskCreate(&hello_task, "hello_task", 2048, NULL, 5, NULL);
	xTaskCreate(&http_server_netconn_thread, "http_server_netconn_thread", 2048,
			NULL, 5, NULL);
}
This gives me some console output, but it hangs on reading.

Code: Select all

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0x00
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3ffc0000,len:0
load:0x3ffc0000,len:920
load:0x40078000,len:2836
ho 0 tail 12 room 4
load:0x40098000,len:692
entry 0x4009813c
I (95) heap_alloc_caps: Initializing heap allocator:
I (95) heap_alloc_caps: Region 19: 3FFC0E88 len 0001F178 tag 0
I (96) heap_alloc_caps: Region 25: 3FFE8000 len 00018000 tag 1
I (106) cpu_start: Pro cpu up.
I (111) cpu_start: Starting app cpu, entry point is 0x40080af0
I (0) cpu_start: App cpu up.
I (126) cpu_start: Pro cpu start user code
rtc v112 Sep 26 2016 22:32:10
XTAL 40M
I (162) cpu_start: Starting scheduler on PRO CPU.
I (42) cpu_start: Starting scheduler on APP CPU.
frc2_timer_task_hdl:3ffc5414, prio:22, stack:2048
tcpip_task_hdlxxx : 3ffc5f74, prio:18,stack:2048
phy_version: 123, Sep 13 2016, 20:01:58, 0
pp_task_hdl : 3ffc97c8, prio:23, stack:8192
I (3412) example: Setting WiFi configuration SSID miau...
mode : sta(24:0a:c4:01:be:f0)
n:1 1, o:1 0, ap:255 255, sta:1 1, prof:1
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0

connected with miau, channel 1
I (11982) event: ip: 192.168.2.111, mask: 255.255.255.0, gw: 192.168.2.1
Connection accepted
About to read from socket

When I connect via netcat:
nc 192.168.2.111 80
Ok, here you go
But I can type and send whatever I want, read() or recv() won't get it.
nc 192.168.2.111 80
Ok, here you go
GET / HTTP/1.0

I suppose for me an working example of listen -> accept -> read with esp32 would be sufficient, so any ideas are welcome.
Thanks.

Re: lwip http server, can't read from socket

Posted: Sun Oct 30, 2016 6:23 pm
by ESP_igrr
I think Sprite has mentioned that he saw some problems with his esphttpd as well, with the latest code. I'll relay your example to our issue tracker in the morning.

Re: lwip http server, can't read from socket

Posted: Sun Oct 30, 2016 8:18 pm
by kolban
Howdy. Not familiar with these "netconn" APIs but I have had good luck using the "sockets" API. Is there a reason that you want to use netconn over sockets?

Re: lwip http server, can't read from socket

Posted: Sun Oct 30, 2016 8:49 pm
by engelant
Hi kolban,
actually I tried using the socket interface before
socket -> bind -> accept
but also was unable to read, it seems to just block and does not return.
If you happen to have an example of how it's supposed to work I'd be happy if you share it.
I'm using the current esp-idf and dependencies from git (last pulled a few hours ago).

Re: lwip http server, can't read from socket

Posted: Mon Oct 31, 2016 6:46 am
by WiFive
Your code works for me.

EDIT: oops you're right it hangs after pulling the latest changes to esp-idf. Works on cbb26c9.

Re: lwip http server, can't read from socket

Posted: Mon Oct 31, 2016 11:13 am
by ESP_Sprite
FYI, we have figured out the issue and a fix should hit master today or tomorrow.

Re: lwip http server, can't read from socket

Posted: Mon Oct 31, 2016 2:41 pm
by kolban
Thankfully I was tinkering with areas outside of networking this weekend but at the end of the day, I was also trying to get an HTTP server working on an ESP32 (happened to be using JavaScript but that is also based on sockets). I found that the code was not seeing any incoming accepts (just as this thread was saying) and had forgotten about this thread. I think what I'm hearing here is that in the ESP-IDF over the weekend, the accept API (or some internals) had stopped working?

In my real-world work, when we have an issue that is liable to affect large numbers of folks, we have a "status" web page that we can go to see if there are "known issues" that I should be cognizant off. Breaks in code happen ... no issues there and ESP-IDF is pre 1.0 and hence all is open season and we have a stable 0.9 as needed.

That said ... I'm wondering if there isn't a place where we can keep a list of "known things that are broken" that are ideally categorized. The github issues list is great for reporting issues ... but that is as much a clearing house and tracking environment as anything and not particularly consumable for "This function is broke".

For my own curiosity, is there an automated regression suite associated with ESP-IDF? If yes, how often is it run? How long does it take to run? Does it produce reports that might form the basis of a "this function failed" report?

Re: lwip http server, can't read from socket

Posted: Mon Oct 31, 2016 3:38 pm
by engelant
ESP_Sprite wrote:FYI, we have figured out the issue and a fix should hit master today or tomorrow.
Thanks for the update.

Re: lwip http server, can't read from socket

Posted: Mon Oct 31, 2016 4:45 pm
by ESP_igrr
Neil,
we have an automated test suite which has two parts:
  • Compilation tests, build system tests, and a few unit tests which can run on the PC. Run for every commit, takes ~5 minutes to run.
  • Integration tests which involve real esp32 devices connected to a PC. Run for each commit on master branch, or, if triggered manually, for any given branch. Full suite takes slightly more than an hour to run (although this can be easily improved because there are many tests and they can run in parallel)
We currently have a lot of tests which are failing because of missing or broken functionality. Normally test failure in master branch would prevent code from being deployed to Github. We have decided that we need to keep delivering previews of new features even if some parts of the system don't work as expected yet. Therefore, as a temporary measure, we push to Github even if something in master branch is broken. QA team is working through test failures, creating issues for failing cases, and is adding them to "known failing" list. This is expressed by adding a hat symbol next the test name in test config files.

The idea about "status page" is a very sound one. By next week we should have an internal dashboard which will show current test status, and progress over time (which test has been failing all the time, which has just started failing). This dashboard can be easily made public, at least for master branch. I'll put this on our QA team plan.

Edit: here's how this status may look (based on the generated test reports):
Screen Shot 2016-11-01 at 12.52.38 AM.png
Screen Shot 2016-11-01 at 12.52.38 AM.png (158.7 KiB) Viewed 19564 times

Re: lwip http server, can't read from socket

Posted: Mon Oct 31, 2016 11:23 pm
by hydrabus
ESP_igrr wrote:Neil,
we have an automated test suite which has two parts:
  • Compilation tests, build system tests, and a few unit tests which can run on the PC. Run for every commit, takes ~5 minutes to run.
  • Integration tests which involve real esp32 devices connected to a PC. Run for each commit on master branch, or, if triggered manually, for any given branch. Full suite takes slightly more than an hour to run (although this can be easily improved because there are many tests and they can run in parallel)
We currently have a lot of tests which are failing because of missing or broken functionality. Normally test failure in master branch would prevent code from being deployed to Github. We have decided that we need to keep delivering previews of new features even if some parts of the system don't work as expected yet. Therefore, as a temporary measure, we push to Github even if something in master branch is broken. QA team is working through test failures, creating issues for failing cases, and is adding them to "known failing" list. This is expressed by adding a hat symbol next the test name in test config files.

The idea about "status page" is a very sound one. By next week we should have an internal dashboard which will show current test status, and progress over time (which test has been failing all the time, which has just started failing). This dashboard can be easily made public, at least for master branch. I'll put this on our QA team plan.

Edit: here's how this status may look (based on the generated test reports):
Screen Shot 2016-11-01 at 12.52.38 AM.png
Thanks to explain and share your internal process
I really like the idea of integration test / automatic tests on PC & real hardware (test driven process) a bit like how it is done for some critical projects in aeronautic (but in your case in a most efficient way) or like explained in Agility Process..., congratulations keep the good work.