Activation functions used in neural networks are often selected based on intuition or dubious theoretical grounding. Biology has long served as an inspiration for many innovations in machine learning, but has had limited success in informing novel activation functions. In this work, we revisit the idea of the sigmoid function, \(S(x) = \frac{1}{1 + e^{-x}}\), and discover that the shape of the similarly-named Sigmoid colon, a portion of the human large intestine, outperforms the traditional sigmoid function. We find a 4% improvement on MNIST, resulting in state of the art performance for gastrointestinally inspired models. Source code for our activation function and tracing tool is available at https://github.com/poopingface/sigmoidcolon.
Coming soon.
Coming soon.
In order to achieve the best performance, we must obtain an accurate 2-dimensional path of the human sigmoid colon. We facilitate this by developing an interactive tool, which allows medical professionals to trace the path of the sigmoid colon on patient MRI scans. The extension of this traced path to cover the domain \((-\infty, \infty)\) will serve as our activation function.
To maintain differentiability and improve run time efficiency, we replace the traced colon function with a polynomial fit. This fit gives us the final equation for our activation function:
$$\sigma_\text{colon}(x) = -0.000198382 * x ^ 3 + 0.133785 * x ^ 2 + -29.0015 * x + 2345.68$$
Coming soon.